[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:37:31 dist-test-slave-tm1g krb5kdc[24359](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:37:31 dist-test-slave-tm1g krb5kdc[24359](info): set up 2 sockets
Aug 06 20:37:31 dist-test-slave-tm1g krb5kdc[24359](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 20:37:32.645515 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:37:32 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:37:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:37:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:37:32.818012 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:32869
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:32869
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:33.166365 24374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:33.166733 24374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:33.166828 24374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:33.174124 24374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:37:33.174228 24374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:33.174276 24374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:33.174402 24374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:37:33.174455 24374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:37:33.184599 24374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:32869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:32869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:33.186133 24374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:33.187546 24374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:33.200587 24383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:33.200626 24379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:33.200672 24381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:33.202005 24374 server_base.cc:1034] running on GCE node
I20260806 20:37:33.202910 24374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:33.204437 24374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:33.205667 24374 hybrid_clock.cc:648] HybridClock initialized: now 1786048653205604 us; error 51 us; skew 500 ppm
I20260806 20:37:33.206029 24374 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:33 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048653, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:33.209327 24374 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:37:33.210901 24374 webserver.cc:460] Webserver started at http://127.23.200.62:33577/ using document root <none> and password file <none>
I20260806 20:37:33.211828 24374 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:33.211984 24374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:33.212299 24374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:33.215821 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "89d502c141b749508a0d8762adb4b671"
format_stamp: "Formatted at 2026-08-06 20:37:33 on dist-test-slave-tm1g"
server_key: "aeb857787f4b20d352704280aaf7775a"
server_key_iv: "0d93075c267de510917c2424cbc43d27"
server_key_version: "encryptionkey@0"
I20260806 20:37:33.216617 24374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "89d502c141b749508a0d8762adb4b671"
format_stamp: "Formatted at 2026-08-06 20:37:33 on dist-test-slave-tm1g"
server_key: "aeb857787f4b20d352704280aaf7775a"
server_key_iv: "0d93075c267de510917c2424cbc43d27"
server_key_version: "encryptionkey@0"
I20260806 20:37:33.222692 24374 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 20:37:33.226540 24390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:33.228031 24374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:37:33.228222 24374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "89d502c141b749508a0d8762adb4b671"
format_stamp: "Formatted at 2026-08-06 20:37:33 on dist-test-slave-tm1g"
server_key: "aeb857787f4b20d352704280aaf7775a"
server_key_iv: "0d93075c267de510917c2424cbc43d27"
server_key_version: "encryptionkey@0"
I20260806 20:37:33.228441 24374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:33.253175 24374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:33.257479 24374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:33.257788 24374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:33.276558 24374 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:32869
I20260806 20:37:33.276592 24442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:32869 every 8 connection(s)
I20260806 20:37:33.278123 24374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:37:33.282562 24443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:33.285166 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 24374
I20260806 20:37:33.285529 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:37:33.287490 24352 external_mini_cluster.cc:1428] Setting key 84927d5255610af9785a68aa80dd5d70
I20260806 20:37:33.295099 24443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671: Bootstrap starting.
I20260806 20:37:33.298205 24443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:33.299270 24443 log.cc:824] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:33.302634 24443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671: No bootstrap required, opened a new log
I20260806 20:37:33.311844 24443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d502c141b749508a0d8762adb4b671" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 32869 } }
I20260806 20:37:33.312316 24443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:33.312439 24443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89d502c141b749508a0d8762adb4b671, State: Initialized, Role: FOLLOWER
I20260806 20:37:33.313109 24443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d502c141b749508a0d8762adb4b671" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 32869 } }
I20260806 20:37:33.313349 24443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:37:33.313505 24443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:37:33.313690 24443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:33.316219 24443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d502c141b749508a0d8762adb4b671" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 32869 } }
I20260806 20:37:33.316807 24443 leader_election.cc:302] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89d502c141b749508a0d8762adb4b671; no voters:
I20260806 20:37:33.317485 24443 leader_election.cc:288] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:37:33.317601 24448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 1 FOLLOWER]: Leader election won for term 1
Aug 06 20:37:33 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:33.319818 24448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [term 1 LEADER]: Becoming Leader. State: Replica: 89d502c141b749508a0d8762adb4b671, State: Running, Role: LEADER
I20260806 20:37:33.319998 24443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:37:33.320569 24448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d502c141b749508a0d8762adb4b671" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 32869 } }
I20260806 20:37:33.324735 24449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89d502c141b749508a0d8762adb4b671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d502c141b749508a0d8762adb4b671" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 32869 } } }
I20260806 20:37:33.325261 24449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:33.324703 24450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89d502c141b749508a0d8762adb4b671. Latest consensus state: current_term: 1 leader_uuid: "89d502c141b749508a0d8762adb4b671" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89d502c141b749508a0d8762adb4b671" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 32869 } } }
I20260806 20:37:33.325881 24450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671 [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:33.336294 24445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:33.291506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.23.200.62:32869)
0806 20:37:33.292706 (+ 1200us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:33.292721 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:33.294095 (+ 1374us) server_negotiation.cc:407] Connection header received
0806 20:37:33.303826 (+ 9731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:33.303853 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:33.304322 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:33.304756 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:33.307663 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:33.309720 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:33.311452 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:33.312527 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:33.312708 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:33.319746 (+ 7038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:33.319758 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:33.319782 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:33.319879 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:33.325965 (+ 6086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:33.327385 (+ 1420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:33.327397 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:33.327409 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:33.327681 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:33.328400 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:33.328408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:33.328414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:33.329412 (+ 998us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:33.333357 (+ 3945us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:33.333687 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 20:37:33.334227 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":601,"threads_started":1}
I20260806 20:37:33.340590 24457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:37:33.345093 24457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:37:33.353942 24457 catalog_manager.cc:1347] Generated new cluster ID: a3b4b4b71dd54a7baf1165c3432ea11a
I20260806 20:37:33.354084 24457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:37:33.361285 24457 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:37:33.362571 24457 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:37:33.379213 24457 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89d502c141b749508a0d8762adb4b671: Generated new TSK 0
I20260806 20:37:33.380013 24457 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:37:33.430023 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:32869
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:33.800972 24471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:33.801231 24471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:33.801407 24471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:33.809649 24471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:33.809782 24471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:33.810048 24471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:37:33.820848 24471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:32869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:33.822196 24471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:33.823568 24471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:33.838768 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:33.839234 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:33.839562 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:33.840231 24471 server_base.cc:1034] running on GCE node
I20260806 20:37:33.840857 24471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:33.842203 24471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:33.843425 24471 hybrid_clock.cc:648] HybridClock initialized: now 1786048653843366 us; error 59 us; skew 500 ppm
I20260806 20:37:33.843731 24471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:33 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048653, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:33.847471 24471 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:37:33.849123 24471 webserver.cc:460] Webserver started at http://127.23.200.1:41273/ using document root <none> and password file <none>
I20260806 20:37:33.849848 24471 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:33.849996 24471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:33.850409 24471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:33.853231 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "4cf95be68cd24a6f959a44e63d100469"
format_stamp: "Formatted at 2026-08-06 20:37:33 on dist-test-slave-tm1g"
server_key: "7ddd79b497370b02ec72b4c5c1830b01"
server_key_iv: "578a97a2456220c41dc1660b0663be5f"
server_key_version: "encryptionkey@0"
I20260806 20:37:33.854032 24471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "4cf95be68cd24a6f959a44e63d100469"
format_stamp: "Formatted at 2026-08-06 20:37:33 on dist-test-slave-tm1g"
server_key: "7ddd79b497370b02ec72b4c5c1830b01"
server_key_iv: "578a97a2456220c41dc1660b0663be5f"
server_key_version: "encryptionkey@0"
I20260806 20:37:33.859603 24471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 20:37:33.863210 24486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:33.864526 24471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:37:33.864701 24471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "4cf95be68cd24a6f959a44e63d100469"
format_stamp: "Formatted at 2026-08-06 20:37:33 on dist-test-slave-tm1g"
server_key: "7ddd79b497370b02ec72b4c5c1830b01"
server_key_iv: "578a97a2456220c41dc1660b0663be5f"
server_key_version: "encryptionkey@0"
I20260806 20:37:33.864893 24471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:33.894609 24471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:33.899621 24471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:33.899992 24471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:33.900988 24471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:33.902724 24471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:33.902825 24471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:33.902973 24471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:33.903045 24471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:33.954432 24471 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:37357
I20260806 20:37:33.954491 24602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:37357 every 8 connection(s)
I20260806 20:37:33.955822 24471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:37:33.957172 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 24471
I20260806 20:37:33.957461 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:37:33.958293 24352 external_mini_cluster.cc:1428] Setting key 57f7539ebd1d2128c6589eefeba9212b
Aug 06 20:37:33 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048653, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:37:33.980939 24605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:33.959273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:42117 (local address 127.23.200.62:32869)
0806 20:37:33.960208 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:33.960212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:33.961338 (+ 1126us) server_negotiation.cc:407] Connection header received
0806 20:37:33.963450 (+ 2112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:33.963459 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:33.963596 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:33.963806 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:33.965282 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:33.966817 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:33.968483 (+ 1666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:33.969359 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:33.969504 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:33.973608 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:33.973617 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:33.973624 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:33.973669 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:33.977590 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:33.978339 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:33.978345 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:33.978350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:33.978525 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:33.979062 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:33.979069 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:33.979073 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:33.979515 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:33.979671 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:33.980473 (+ 802us) server_negotiation.cc:299] Negotiation successful
0806 20:37:33.980641 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":347,"threads_started":1}
I20260806 20:37:33.981791 24606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:33.960082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:32869 (local address 127.23.200.1:42117)
0806 20:37:33.961158 (+ 1076us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:33.961206 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:33.963155 (+ 1949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:33.963945 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:33.963953 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:33.964481 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:33.965037 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:33.965055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:33.967023 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:33.967027 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:33.968338 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:33.968350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:33.969558 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:33.969563 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:33.969873 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:33.970700 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:33.970722 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:33.973377 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:33.977764 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:33.977770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:33.977783 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:33.978188 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:33.978656 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:33.978663 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:33.978666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:33.978922 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:33.979657 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:33.979663 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:33.980001 (+ 338us) client_negotiation.cc:769] Sending connection context
0806 20:37:33.980281 (+ 280us) client_negotiation.cc:240] Negotiation successful
0806 20:37:33.980659 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"thread_start_us":475,"threads_started":1}
I20260806 20:37:33.983366 24603 heartbeater.cc:347] Connected to a master server at 127.23.200.62:32869
I20260806 20:37:33.983721 24603 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:33.984694 24603 heartbeater.cc:511] Master 127.23.200.62:32869 requested a full tablet report, sending...
I20260806 20:37:33.986886 24407 ts_manager.cc:194] Registered new tserver with Master: 4cf95be68cd24a6f959a44e63d100469 (127.23.200.1:37357)
I20260806 20:37:33.988473 24407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:42117
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
W20260806 20:37:34.759176 24352 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.23.200.2: real 0.798s user 0.000s sys 0.012s
I20260806 20:37:34.760052 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:32869
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:37:34.991283 24603 heartbeater.cc:503] Master 127.23.200.62:32869 was elected leader, sending a full tablet report...
W20260806 20:37:35.183544 24609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:35.183871 24609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:35.184054 24609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:35.192059 24609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:35.192209 24609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:35.192399 24609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:37:35.203227 24609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:32869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:35.204551 24609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:35.205931 24609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:35.220355 24616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:35.220402 24617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:35.220448 24619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:35.221304 24609 server_base.cc:1034] running on GCE node
I20260806 20:37:35.221998 24609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:35.223141 24609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:35.224380 24609 hybrid_clock.cc:648] HybridClock initialized: now 1786048655224337 us; error 61 us; skew 500 ppm
I20260806 20:37:35.224709 24609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:35 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:35.232424 24609 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:37:35.234359 24609 webserver.cc:460] Webserver started at http://127.23.200.2:42665/ using document root <none> and password file <none>
I20260806 20:37:35.235085 24609 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:35.235239 24609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:35.235627 24609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:35.239037 24609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "8892ab31287d455d9ee9f8358ba57943"
format_stamp: "Formatted at 2026-08-06 20:37:35 on dist-test-slave-tm1g"
server_key: "6c2d1e4fc4979cf1b916bfed5d191979"
server_key_iv: "95ce5db9dfce899810c35487ea4c3433"
server_key_version: "encryptionkey@0"
I20260806 20:37:35.239869 24609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "8892ab31287d455d9ee9f8358ba57943"
format_stamp: "Formatted at 2026-08-06 20:37:35 on dist-test-slave-tm1g"
server_key: "6c2d1e4fc4979cf1b916bfed5d191979"
server_key_iv: "95ce5db9dfce899810c35487ea4c3433"
server_key_version: "encryptionkey@0"
I20260806 20:37:35.245684 24609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:37:35.249341 24626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:35.250710 24609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 20:37:35.250881 24609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "8892ab31287d455d9ee9f8358ba57943"
format_stamp: "Formatted at 2026-08-06 20:37:35 on dist-test-slave-tm1g"
server_key: "6c2d1e4fc4979cf1b916bfed5d191979"
server_key_iv: "95ce5db9dfce899810c35487ea4c3433"
server_key_version: "encryptionkey@0"
I20260806 20:37:35.251075 24609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:35.285053 24609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:35.289256 24609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:35.289563 24609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:35.290582 24609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:35.292130 24609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:35.292238 24609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:35.292399 24609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:35.292485 24609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:35.342510 24609 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:43491
I20260806 20:37:35.342599 24744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:43491 every 8 connection(s)
I20260806 20:37:35.343902 24609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:37:35.350005 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 24609
I20260806 20:37:35.350217 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:37:35.350831 24352 external_mini_cluster.cc:1428] Setting key 46073465eebdb6db933c95c777333353
Aug 06 20:37:35 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:35.370463 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:35.346619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:50837 (local address 127.23.200.62:32869)
0806 20:37:35.347097 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:35.347101 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:35.348119 (+ 1018us) server_negotiation.cc:407] Connection header received
0806 20:37:35.350320 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:35.350328 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:35.350434 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:35.350560 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:35.352318 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.353523 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.356859 (+ 3336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.357279 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.357380 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:35.362531 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:35.362540 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:35.362548 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:35.362595 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:35.366695 (+ 4100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:35.367426 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:35.367448 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:35.367454 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:35.367632 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:35.368094 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:35.368101 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:35.368105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:35.368566 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:35.368754 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:35.369629 (+ 875us) server_negotiation.cc:299] Negotiation successful
0806 20:37:35.369883 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260806 20:37:35.371362 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:35.347181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:32869 (local address 127.23.200.2:50837)
0806 20:37:35.347942 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:35.347991 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:35.349929 (+ 1938us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:35.350766 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:35.350774 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:35.351371 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:35.352131 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:35.352152 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.353699 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.353707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:35.356655 (+ 2948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:35.356680 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.357591 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.357602 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:35.357966 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:35.358885 (+ 919us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:35.358924 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:35.362218 (+ 3294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:35.366841 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:35.366849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:35.366863 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:35.367300 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:35.367768 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:35.367771 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:35.367773 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:35.367972 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:35.368723 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:35.368730 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:35.369179 (+ 449us) client_negotiation.cc:769] Sending connection context
0806 20:37:35.369432 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 20:37:35.369954 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":272,"threads_started":1}
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
I20260806 20:37:35.373096 24745 heartbeater.cc:347] Connected to a master server at 127.23.200.62:32869
I20260806 20:37:35.373581 24745 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:35.374562 24745 heartbeater.cc:511] Master 127.23.200.62:32869 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
I20260806 20:37:35.376302 24407 ts_manager.cc:194] Registered new tserver with Master: 8892ab31287d455d9ee9f8358ba57943 (127.23.200.2:43491)
I20260806 20:37:35.377158 24407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:50837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:37:35.396991 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:32869
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:35.807073 24751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:35.807469 24751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:35.807679 24751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:35.815582 24751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:35.815722 24751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:35.815919 24751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:37:35.826934 24751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:32869
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:35.828315 24751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:35.829666 24751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:35.844713 24756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:35.844908 24757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:35.845001 24759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:35.845960 24751 server_base.cc:1034] running on GCE node
I20260806 20:37:35.846565 24751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:35.847417 24751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:35.848611 24751 hybrid_clock.cc:648] HybridClock initialized: now 1786048655848557 us; error 45 us; skew 500 ppm
I20260806 20:37:35.848953 24751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:35 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:35.852475 24751 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:37:35.854179 24751 webserver.cc:460] Webserver started at http://127.23.200.3:40511/ using document root <none> and password file <none>
I20260806 20:37:35.854964 24751 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:35.855098 24751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:35.855425 24751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:35.858258 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "087d50b6139040bfb687b3c802379458"
format_stamp: "Formatted at 2026-08-06 20:37:35 on dist-test-slave-tm1g"
server_key: "e5406788802a3806d5effe69dab0ab2e"
server_key_iv: "3824d74bf1a81ba23f2c55b559d953bb"
server_key_version: "encryptionkey@0"
I20260806 20:37:35.859035 24751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "087d50b6139040bfb687b3c802379458"
format_stamp: "Formatted at 2026-08-06 20:37:35 on dist-test-slave-tm1g"
server_key: "e5406788802a3806d5effe69dab0ab2e"
server_key_iv: "3824d74bf1a81ba23f2c55b559d953bb"
server_key_version: "encryptionkey@0"
I20260806 20:37:35.864462 24751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:37:35.868224 24767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:35.869658 24751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:37:35.869788 24751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "087d50b6139040bfb687b3c802379458"
format_stamp: "Formatted at 2026-08-06 20:37:35 on dist-test-slave-tm1g"
server_key: "e5406788802a3806d5effe69dab0ab2e"
server_key_iv: "3824d74bf1a81ba23f2c55b559d953bb"
server_key_version: "encryptionkey@0"
I20260806 20:37:35.869959 24751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:35.889678 24751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:35.894104 24751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:35.894539 24751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:35.895531 24751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:35.897018 24751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:35.897184 24751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:35.897399 24751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:35.897534 24751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:35.948194 24751 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:42697
I20260806 20:37:35.948292 24884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:42697 every 8 connection(s)
I20260806 20:37:35.949572 24751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:37:35.954391 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 24751
I20260806 20:37:35.954563 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:37:35.955196 24352 external_mini_cluster.cc:1428] Setting key cf6a4da2aa00122cffc5d443f09a8104
Aug 06 20:37:35 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048655, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:35.972141 24888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:35.952390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:48715 (local address 127.23.200.62:32869)
0806 20:37:35.952939 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:35.952944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:35.954385 (+ 1441us) server_negotiation.cc:407] Connection header received
0806 20:37:35.956754 (+ 2369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:35.956760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:35.956863 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:35.956977 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:35.959259 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.960070 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.961743 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.962132 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.962227 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:35.965870 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:35.965875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:35.965878 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:35.965909 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:35.968610 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:35.969466 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:35.969471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:35.969474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:35.969640 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:35.970208 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:35.970213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:35.970215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:35.970610 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:35.970772 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:35.971638 (+ 866us) server_negotiation.cc:299] Negotiation successful
0806 20:37:35.971802 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":314,"threads_started":1}
I20260806 20:37:35.972934 24889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:35.953052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:32869 (local address 127.23.200.3:48715)
0806 20:37:35.954054 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:35.954104 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:35.956449 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:35.957343 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:35.957357 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:35.958283 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:35.959052 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:35.959071 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.960200 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.960204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:35.961621 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:35.961633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:35.962435 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:35.962442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:35.962690 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:35.963405 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:35.963426 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:35.965687 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:35.968730 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:35.968736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:35.968747 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:35.969330 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:35.969779 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:35.969783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:35.969786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:35.970070 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:35.970791 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:35.970802 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:35.971201 (+ 399us) client_negotiation.cc:769] Sending connection context
0806 20:37:35.971531 (+ 330us) client_negotiation.cc:240] Negotiation successful
0806 20:37:35.971886 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":441,"threads_started":1}
I20260806 20:37:35.974437 24885 heartbeater.cc:347] Connected to a master server at 127.23.200.62:32869
I20260806 20:37:35.974766 24885 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:35.975621 24885 heartbeater.cc:511] Master 127.23.200.62:32869 requested a full tablet report, sending...
I20260806 20:37:35.977022 24407 ts_manager.cc:194] Registered new tserver with Master: 087d50b6139040bfb687b3c802379458 (127.23.200.3:42697)
I20260806 20:37:35.977815 24407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:48715
I20260806 20:37:35.983315 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:36.030608 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:36.013867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38404 (local address 127.23.200.1:37357)
0806 20:37:36.014784 (+ 917us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:36.014795 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:36.015164 (+ 369us) server_negotiation.cc:407] Connection header received
0806 20:37:36.015318 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:36.015326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:36.015564 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:36.015691 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:36.017153 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.018135 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.019645 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.020229 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.020447 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:36.024769 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:36.024780 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:36.024793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:36.024830 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:36.027876 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.028550 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.028557 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.028559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.028679 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.029043 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.029046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.029048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.029380 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:36.029614 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:36.029846 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 20:37:36.030084 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":488,"threads_started":1}
W20260806 20:37:36.032054 24516 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38404
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24359](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:36.077693 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:36.063746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38410 (local address 127.23.200.1:37357)
0806 20:37:36.064018 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:36.064025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:36.064223 (+ 198us) server_negotiation.cc:407] Connection header received
0806 20:37:36.064375 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:36.064379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:36.064478 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:36.064565 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:36.065897 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.066643 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.067814 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.068262 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.068357 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:36.072585 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:36.072590 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:36.072593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:36.072625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:36.075472 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.076075 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.076079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.076082 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.076256 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.076555 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.076558 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.076560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.076828 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:36.077071 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:36.077222 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 20:37:36.077375 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 20:37:36.079380 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 24471
I20260806 20:37:36.092052 24894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:36.080642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.23.200.1:37357)
0806 20:37:36.080832 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:36.080838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:36.080855 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:37:36.081001 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:36.081008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:36.081074 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:36.081175 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:36.082276 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.082997 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.084270 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.084732 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.084870 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:36.087022 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:36.087027 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:36.087031 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:36.087062 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:36.089354 (+ 2292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.089967 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.089974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.089978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.090180 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.090581 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.090588 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.090595 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.090905 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:36.091173 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:36.091460 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 20:37:36.091672 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 20:37:36.092893 24598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:36.179064 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 24471
I20260806 20:37:36.197387 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 24609
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:37:36.213127 24901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:36.198411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60102 (local address 127.23.200.2:43491)
0806 20:37:36.198955 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:36.198962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:36.198989 (+ 27us) server_negotiation.cc:407] Connection header received
0806 20:37:36.199049 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:36.199055 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:36.199257 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:36.199447 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:36.200812 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.201819 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.203113 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.203630 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.203725 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:36.207614 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:36.207625 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:36.207637 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:36.207671 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:36.210587 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.211169 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.211175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.211178 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.211302 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.211690 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.211694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.211697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.212044 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:36.212251 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:36.212528 (+ 277us) server_negotiation.cc:299] Negotiation successful
0806 20:37:36.212742 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":274,"threads_started":1}
I20260806 20:37:36.214624 24739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:36.297981 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 24609
I20260806 20:37:36.315279 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 24751
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24359](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048656, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:37:36.331135 24903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:36.316327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55910 (local address 127.23.200.3:42697)
0806 20:37:36.316950 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:36.316957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:36.316986 (+ 29us) server_negotiation.cc:407] Connection header received
0806 20:37:36.317050 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:36.317056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:36.317268 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:36.317394 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:36.318685 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.319723 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.320983 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:36.321597 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:36.321695 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:36.325850 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:36.325861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:36.325875 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:36.325911 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:36.328603 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.329183 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.329190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.329192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.329313 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:36.329633 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:36.329636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:36.329638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:36.329951 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:36.330192 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:36.330539 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 20:37:36.330738 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":313,"threads_started":1}
I20260806 20:37:36.332543 24880 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:36.414721 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 24751
I20260806 20:37:36.432020 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 24374
I20260806 20:37:36.433218 24438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:36.537987 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 24374
2026-08-06T20:37:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5020 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24908](info): set up 2 sockets
Aug 06 20:37:36 dist-test-slave-tm1g krb5kdc[24908](info): commencing operation
krb5kdc: starting...
W20260806 20:37:37.633688 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.009s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:37:37 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:37:37.818202 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:38713
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:38713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:38.136014 24921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:38.136281 24921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:38.136456 24921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:38.143356 24921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:37:38.143457 24921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:38.143548 24921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:38.143651 24921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:37:38.143713 24921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:37:38.153272 24921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:38713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:38713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:38.154623 24921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:38.155937 24921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:38.167650 24927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:38.167920 24928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:38.168231 24930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:38.168902 24921 server_base.cc:1034] running on GCE node
I20260806 20:37:38.169697 24921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:38.171150 24921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:38.172363 24921 hybrid_clock.cc:648] HybridClock initialized: now 1786048658172320 us; error 31 us; skew 500 ppm
I20260806 20:37:38.172705 24921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:38 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048658, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:38.175874 24921 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:37:38.177367 24921 webserver.cc:460] Webserver started at http://127.23.200.62:46415/ using document root <none> and password file <none>
I20260806 20:37:38.178071 24921 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:38.178217 24921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:38.178591 24921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:38.181497 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "52e96c1597f84df29b0b62142f38e47d"
format_stamp: "Formatted at 2026-08-06 20:37:38 on dist-test-slave-tm1g"
server_key: "ac68df4ed2a39f4e3db1ad91efd2a7b1"
server_key_iv: "dfae3ce6adb69a17162d483627993eba"
server_key_version: "encryptionkey@0"
I20260806 20:37:38.182247 24921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "52e96c1597f84df29b0b62142f38e47d"
format_stamp: "Formatted at 2026-08-06 20:37:38 on dist-test-slave-tm1g"
server_key: "ac68df4ed2a39f4e3db1ad91efd2a7b1"
server_key_iv: "dfae3ce6adb69a17162d483627993eba"
server_key_version: "encryptionkey@0"
I20260806 20:37:38.187325 24921 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:37:38.190723 24937 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:38.192132 24921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:37:38.192313 24921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "52e96c1597f84df29b0b62142f38e47d"
format_stamp: "Formatted at 2026-08-06 20:37:38 on dist-test-slave-tm1g"
server_key: "ac68df4ed2a39f4e3db1ad91efd2a7b1"
server_key_iv: "dfae3ce6adb69a17162d483627993eba"
server_key_version: "encryptionkey@0"
I20260806 20:37:38.192512 24921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:38.212586 24921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:38.217017 24921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:38.217304 24921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:38.236078 24921 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:38713
I20260806 20:37:38.236109 24992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:38713 every 8 connection(s)
I20260806 20:37:38.237422 24921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:37:38.241067 24993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:38.243553 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 24921
I20260806 20:37:38.243764 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:37:38.244560 24352 external_mini_cluster.cc:1428] Setting key 8642f564f889b564179b87bbc5f88d9b
I20260806 20:37:38.253401 24993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d: Bootstrap starting.
I20260806 20:37:38.257205 24993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:38.258564 24993 log.cc:824] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d: Log is configured to *not* fsync() on all Append() calls
Aug 06 20:37:38 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:38.261389 24993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d: No bootstrap required, opened a new log
I20260806 20:37:38.268115 24996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:38.246797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49538 (local address 127.23.200.62:38713)
0806 20:37:38.247566 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:38.247577 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:38.247617 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:37:38.249228 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:38.249247 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:38.249620 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:38.249932 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:38.251865 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:38.253166 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:38.254901 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:38.255570 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:38.255721 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:38.260716 (+ 4995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:38.260723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:38.260737 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:38.260771 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:38.263914 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:38.264615 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:38.264621 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:38.264630 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:38.264791 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:38.265198 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:38.265214 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:38.265217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:38.265775 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:38.266098 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:38.266683 (+ 585us) server_negotiation.cc:299] Negotiation successful
0806 20:37:38.267092 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":377,"threads_started":1}
I20260806 20:37:38.269255 24993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e96c1597f84df29b0b62142f38e47d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38713 } }
I20260806 20:37:38.269661 24993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:38.269798 24993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52e96c1597f84df29b0b62142f38e47d, State: Initialized, Role: FOLLOWER
I20260806 20:37:38.270448 24993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e96c1597f84df29b0b62142f38e47d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38713 } }
I20260806 20:37:38.270704 24993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:37:38.270854 24993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:37:38.271070 24993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:38.273156 24993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e96c1597f84df29b0b62142f38e47d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38713 } }
I20260806 20:37:38.273646 24993 leader_election.cc:302] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52e96c1597f84df29b0b62142f38e47d; no voters:
I20260806 20:37:38.274196 24993 leader_election.cc:288] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:37:38.274349 24998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:38.275786 24998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [term 1 LEADER]: Becoming Leader. State: Replica: 52e96c1597f84df29b0b62142f38e47d, State: Running, Role: LEADER
I20260806 20:37:38.276273 24993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:37:38.276310 24998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e96c1597f84df29b0b62142f38e47d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38713 } }
I20260806 20:37:38.279259 24999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52e96c1597f84df29b0b62142f38e47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e96c1597f84df29b0b62142f38e47d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38713 } } }
I20260806 20:37:38.279723 24999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:38.279965 25000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52e96c1597f84df29b0b62142f38e47d. Latest consensus state: current_term: 1 leader_uuid: "52e96c1597f84df29b0b62142f38e47d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52e96c1597f84df29b0b62142f38e47d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38713 } } }
I20260806 20:37:38.280315 25000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:38.281412 25007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:37:38.286146 25007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:37:38.295825 25007 catalog_manager.cc:1347] Generated new cluster ID: 36a036d2209e47c3a664f2b84ef14ef9
I20260806 20:37:38.295943 25007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:37:38.305667 25007 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:37:38.307008 25007 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:37:38.317812 25007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52e96c1597f84df29b0b62142f38e47d: Generated new TSK 0
I20260806 20:37:38.318702 25007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:37:38.363687 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38713
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:38.695119 25020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:38.695441 25020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:38.695575 25020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:38.702996 25020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:38.703100 25020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:38.703280 25020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:37:38.712951 25020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:38.714283 25020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:38.715699 25020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:38.729039 25027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:38.729182 25029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:38.729566 25026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:38.729984 25020 server_base.cc:1034] running on GCE node
I20260806 20:37:38.730656 25020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:38.731564 25020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:38.732800 25020 hybrid_clock.cc:648] HybridClock initialized: now 1786048658732753 us; error 53 us; skew 500 ppm
I20260806 20:37:38.733110 25020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:38 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048658, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:38.736346 25020 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:37:38.737792 25020 webserver.cc:460] Webserver started at http://127.23.200.1:44891/ using document root <none> and password file <none>
I20260806 20:37:38.738515 25020 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:38.738629 25020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:38.738896 25020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:38.741876 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "2396d13124034f7e9f21f9708b6deeb8"
format_stamp: "Formatted at 2026-08-06 20:37:38 on dist-test-slave-tm1g"
server_key: "e0c8a5fc644337a06aa6e7136a8a8f96"
server_key_iv: "2deab4fdab5beadc0a9b24d00cf96189"
server_key_version: "encryptionkey@0"
I20260806 20:37:38.742691 25020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "2396d13124034f7e9f21f9708b6deeb8"
format_stamp: "Formatted at 2026-08-06 20:37:38 on dist-test-slave-tm1g"
server_key: "e0c8a5fc644337a06aa6e7136a8a8f96"
server_key_iv: "2deab4fdab5beadc0a9b24d00cf96189"
server_key_version: "encryptionkey@0"
I20260806 20:37:38.747581 25020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 20:37:38.750970 25037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:38.752234 25020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:37:38.752369 25020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "2396d13124034f7e9f21f9708b6deeb8"
format_stamp: "Formatted at 2026-08-06 20:37:38 on dist-test-slave-tm1g"
server_key: "e0c8a5fc644337a06aa6e7136a8a8f96"
server_key_iv: "2deab4fdab5beadc0a9b24d00cf96189"
server_key_version: "encryptionkey@0"
I20260806 20:37:38.752609 25020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:38.781399 25020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:38.785898 25020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:38.786203 25020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:38.787175 25020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:38.788479 25020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:38.788553 25020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:38.788718 25020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:38.788779 25020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:38.822134 25020 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:46869
I20260806 20:37:38.822225 25154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:46869 every 8 connection(s)
I20260806 20:37:38.823509 25020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:37:38.831983 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25020
I20260806 20:37:38.832206 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:37:38.832891 24352 external_mini_cluster.cc:1428] Setting key cae28fd64e691d8a408ccd3940a0a5bc
Aug 06 20:37:38 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048658, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:38.844343 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:38.826139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:34893 (local address 127.23.200.62:38713)
0806 20:37:38.826603 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:38.826608 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:38.827651 (+ 1043us) server_negotiation.cc:407] Connection header received
0806 20:37:38.829484 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:38.829490 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:38.829590 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:38.829712 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:38.831267 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:38.832046 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:38.833788 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:38.834399 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:38.834531 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:38.838698 (+ 4167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:38.838704 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:38.838707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:38.838737 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:38.841268 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:38.842010 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:38.842015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:38.842017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:38.842129 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:38.842629 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:38.842634 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:38.842639 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:38.842893 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:38.843054 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:38.843750 (+ 696us) server_negotiation.cc:299] Negotiation successful
0806 20:37:38.843982 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":252,"threads_started":1}
I20260806 20:37:38.845351 25158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:38.826802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38713 (local address 127.23.200.1:34893)
0806 20:37:38.827498 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:38.827537 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:38.829212 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:38.829946 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:38.829954 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:38.830539 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:38.831099 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:38.831117 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:38.832196 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:38.832204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:38.833639 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:38.833658 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:38.834637 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:38.834646 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:38.834959 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:38.835696 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:38.835729 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:38.838483 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:38.841413 (+ 2930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:38.841422 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:38.841439 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:38.841868 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:38.842242 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:38.842247 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:38.842250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:38.842510 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:38.843036 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:38.843044 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:38.843381 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 20:37:38.843638 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 20:37:38.844151 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":266,"threads_started":1}
I20260806 20:37:38.847065 25155 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38713
I20260806 20:37:38.847374 25155 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:38.848097 25155 heartbeater.cc:511] Master 127.23.200.62:38713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:37:38.850561 24954 ts_manager.cc:194] Registered new tserver with Master: 2396d13124034f7e9f21f9708b6deeb8 (127.23.200.1:46869)
I20260806 20:37:38.852293 24954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:34893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:37:38.875209 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38713
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:39.196492 25161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:39.196762 25161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:39.196938 25161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:39.204195 25161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:39.204296 25161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:39.204509 25161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:37:39.214365 25161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:39.215700 25161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:39.217074 25161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:39.230484 25166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:39.230439 25169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:39.230551 25167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:39.231650 25161 server_base.cc:1034] running on GCE node
I20260806 20:37:39.232259 25161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:39.233127 25161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:39.234378 25161 hybrid_clock.cc:648] HybridClock initialized: now 1786048659234278 us; error 40 us; skew 500 ppm
I20260806 20:37:39.234691 25161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:39 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048659, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:39.242201 25161 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:37:39.243772 25161 webserver.cc:460] Webserver started at http://127.23.200.2:33045/ using document root <none> and password file <none>
I20260806 20:37:39.244483 25161 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:39.244632 25161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:39.244949 25161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:39.247874 25161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "21f76d5481e24cd4a14ce070c1e417bf"
format_stamp: "Formatted at 2026-08-06 20:37:39 on dist-test-slave-tm1g"
server_key: "c750b45605b7101adfab54fe8c1752fb"
server_key_iv: "59d8d053b1984f5d084e54019595cf14"
server_key_version: "encryptionkey@0"
I20260806 20:37:39.248652 25161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "21f76d5481e24cd4a14ce070c1e417bf"
format_stamp: "Formatted at 2026-08-06 20:37:39 on dist-test-slave-tm1g"
server_key: "c750b45605b7101adfab54fe8c1752fb"
server_key_iv: "59d8d053b1984f5d084e54019595cf14"
server_key_version: "encryptionkey@0"
I20260806 20:37:39.253774 25161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:37:39.257319 25176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:39.258654 25161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:39.258833 25161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "21f76d5481e24cd4a14ce070c1e417bf"
format_stamp: "Formatted at 2026-08-06 20:37:39 on dist-test-slave-tm1g"
server_key: "c750b45605b7101adfab54fe8c1752fb"
server_key_iv: "59d8d053b1984f5d084e54019595cf14"
server_key_version: "encryptionkey@0"
I20260806 20:37:39.259032 25161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:39.276247 25161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:39.280292 25161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:39.280592 25161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:39.281601 25161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:39.283033 25161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:39.283106 25161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:39.283233 25161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:39.283283 25161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:39.315243 25161 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33953
I20260806 20:37:39.315285 25294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33953 every 8 connection(s)
I20260806 20:37:39.316550 25161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:37:39.321751 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25161
I20260806 20:37:39.321938 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:37:39.322554 24352 external_mini_cluster.cc:1428] Setting key ed7a9e7c2f9d3a30f5817ed4a63d78d1
Aug 06 20:37:39 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048659, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:39.337993 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.319249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:52807 (local address 127.23.200.62:38713)
0806 20:37:39.319430 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:39.319435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:39.320606 (+ 1171us) server_negotiation.cc:407] Connection header received
0806 20:37:39.322624 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:39.322630 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:39.322722 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:39.322835 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:39.324281 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.324998 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.327646 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.328080 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.328174 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.332276 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:39.332281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:39.332285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:39.332316 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:39.334987 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:39.335713 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:39.335721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:39.335729 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:39.335863 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:39.336313 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:39.336317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:39.336319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:39.336585 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:39.336747 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:39.337412 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 20:37:39.337656 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:37:39.339030 25297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.319746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38713 (local address 127.23.200.2:52807)
0806 20:37:39.320430 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:39.320467 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:39.322362 (+ 1895us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:39.323071 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:39.323080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:39.323574 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:39.324110 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.324127 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.325206 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.325214 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.327480 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.327518 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.328305 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.328314 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.328677 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.329369 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:39.329404 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:39.332042 (+ 2638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:39.335134 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:39.335143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:39.335159 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:39.335581 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:39.335979 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:39.335984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:39.335988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:39.336181 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:39.336719 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:39.336726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:39.337053 (+ 327us) client_negotiation.cc:769] Sending connection context
0806 20:37:39.337310 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 20:37:39.337831 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":277,"threads_started":1}
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
I20260806 20:37:39.340617 25295 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38713
I20260806 20:37:39.340930 25295 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:39.341656 25295 heartbeater.cc:511] Master 127.23.200.62:38713 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
I20260806 20:37:39.343114 24954 ts_manager.cc:194] Registered new tserver with Master: 21f76d5481e24cd4a14ce070c1e417bf (127.23.200.2:33953)
I20260806 20:37:39.343909 24954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:52807
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:37:39.367836 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38713
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:39.696689 25300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:39.696943 25300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:39.697126 25300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:39.704749 25300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:39.704844 25300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:39.705036 25300 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:37:39.714455 25300 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:39.715654 25300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:39.716967 25300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:39.729943 25306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:39.729995 25309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:39.730031 25307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:39.731741 25300 server_base.cc:1034] running on GCE node
I20260806 20:37:39.732342 25300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:39.733227 25300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:39.734453 25300 hybrid_clock.cc:648] HybridClock initialized: now 1786048659734404 us; error 49 us; skew 500 ppm
I20260806 20:37:39.734776 25300 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:39 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048659, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:39.737756 25300 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:37:39.739205 25300 webserver.cc:460] Webserver started at http://127.23.200.3:37557/ using document root <none> and password file <none>
I20260806 20:37:39.740108 25300 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:39.740262 25300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:39.740566 25300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:39.743430 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "d4e9493efff0475198b4dd45de736ce2"
format_stamp: "Formatted at 2026-08-06 20:37:39 on dist-test-slave-tm1g"
server_key: "6fe753d3b3d3cf834fb3bc2a0b7ede68"
server_key_iv: "b4156a075e6af298b983fa25faa184f5"
server_key_version: "encryptionkey@0"
I20260806 20:37:39.744182 25300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "d4e9493efff0475198b4dd45de736ce2"
format_stamp: "Formatted at 2026-08-06 20:37:39 on dist-test-slave-tm1g"
server_key: "6fe753d3b3d3cf834fb3bc2a0b7ede68"
server_key_iv: "b4156a075e6af298b983fa25faa184f5"
server_key_version: "encryptionkey@0"
I20260806 20:37:39.750728 25300 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 20:37:39.754015 25317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:39.755451 25300 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:39.755636 25300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "d4e9493efff0475198b4dd45de736ce2"
format_stamp: "Formatted at 2026-08-06 20:37:39 on dist-test-slave-tm1g"
server_key: "6fe753d3b3d3cf834fb3bc2a0b7ede68"
server_key_iv: "b4156a075e6af298b983fa25faa184f5"
server_key_version: "encryptionkey@0"
I20260806 20:37:39.755851 25300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:39.780422 25300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:39.784687 25300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:39.784969 25300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:39.785892 25300 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:39.787276 25300 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:39.787372 25300 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:39.787528 25300 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:39.787599 25300 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:39.819398 25300 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:35297
I20260806 20:37:39.819574 25437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:35297 every 8 connection(s)
I20260806 20:37:39.820744 25300 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:37:39.828126 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25300
I20260806 20:37:39.828311 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:37:39.828948 24352 external_mini_cluster.cc:1428] Setting key 45cd79f999f9e5a9659996002154f442
Aug 06 20:37:39 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048659, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:39.843186 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.823739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:55763 (local address 127.23.200.62:38713)
0806 20:37:39.823955 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:39.823962 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:39.825400 (+ 1438us) server_negotiation.cc:407] Connection header received
0806 20:37:39.828083 (+ 2683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:39.828091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:39.828187 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:39.828303 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:39.830432 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.831314 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.833122 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.833553 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.833682 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.837452 (+ 3770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:39.837457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:39.837460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:39.837489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:39.840175 (+ 2686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:39.840854 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:39.840859 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:39.840861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:39.840984 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:39.841408 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:39.841411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:39.841414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:39.841717 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:39.841869 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:39.842616 (+ 747us) server_negotiation.cc:299] Negotiation successful
0806 20:37:39.842836 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:37:39.844224 25440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.824257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38713 (local address 127.23.200.3:55763)
0806 20:37:39.825205 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:39.825262 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:39.827739 (+ 2477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:39.828731 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:39.828742 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:39.829517 (+ 775us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:39.830206 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.830250 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.831470 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.831476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.832966 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.832995 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.833841 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.833850 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.834183 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.834951 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:39.834971 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:39.837257 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:39.840311 (+ 3054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:39.840319 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:39.840333 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:39.840726 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:39.841097 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:39.841102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:39.841106 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:39.841276 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:39.841843 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:39.841850 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:39.842168 (+ 318us) client_negotiation.cc:769] Sending connection context
0806 20:37:39.842458 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 20:37:39.842951 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":452,"threads_started":1}
I20260806 20:37:39.845544 25438 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38713
I20260806 20:37:39.845842 25438 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:39.846657 25438 heartbeater.cc:511] Master 127.23.200.62:38713 requested a full tablet report, sending...
I20260806 20:37:39.848006 24954 ts_manager.cc:194] Registered new tserver with Master: d4e9493efff0475198b4dd45de736ce2 (127.23.200.3:35297)
I20260806 20:37:39.848721 24954 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:55763
I20260806 20:37:39.855371 25155 heartbeater.cc:503] Master 127.23.200.62:38713 was elected leader, sending a full tablet report...
I20260806 20:37:39.856909 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:37:39.882279 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.865436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44698 (local address 127.23.200.62:38713)
0806 20:37:39.865661 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:39.865667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:39.866467 (+ 800us) server_negotiation.cc:407] Connection header received
0806 20:37:39.866654 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:39.866657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:39.866714 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:39.866791 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:39.869476 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.870581 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.872586 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.873051 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.873188 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.876211 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:39.876217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:39.876222 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:39.876261 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:39.879520 (+ 3259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:39.880191 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:39.880199 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:39.880202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:39.880336 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:39.880783 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:39.880789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:39.880792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:39.881133 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:39.881403 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:39.881746 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 20:37:39.881962 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 20:37:39.888561 24954 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44698:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:37:39.896256 24954 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:39.923226 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.911199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44032 (local address 127.23.200.1:46869)
0806 20:37:39.913003 (+ 1804us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:39.913015 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:39.914207 (+ 1192us) server_negotiation.cc:407] Connection header received
0806 20:37:39.914333 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:39.914342 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:39.914570 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:39.914762 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:39.915760 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.917692 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.920284 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.921790 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.921962 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.922213 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:39.922480 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 20:37:39.922660 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1617,"thread_start_us":1427,"threads_started":1}
I20260806 20:37:39.923242 25453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.910288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:46869 (local address 127.0.0.1:44032)
0806 20:37:39.914019 (+ 3731us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:39.914035 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:39.914150 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:39.915035 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:39.915039 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:39.915082 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:39.915524 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.915541 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.917868 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.917873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.920128 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.920140 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.921993 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.922023 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.922204 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.922253 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 20:37:39.922449 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 20:37:39.922647 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"mutex_wait_us":60,"thread_start_us":3275,"threads_started":1}
I20260806 20:37:39.924337 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.909128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33953 (local address 127.0.0.1:39600)
0806 20:37:39.911775 (+ 2647us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:39.911819 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:39.911978 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:39.915584 (+ 3606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:39.915589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:39.915614 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:39.915991 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.916000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.918067 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.918072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.920941 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.920956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.923097 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.923106 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.923360 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.923392 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 20:37:39.923455 (+ 63us) client_negotiation.cc:240] Negotiation successful
0806 20:37:39.923552 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2425,"mutex_wait_us":6,"thread_start_us":2252,"threads_started":1}
I20260806 20:37:39.925146 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.909406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:35297 (local address 127.0.0.1:34678)
0806 20:37:39.912642 (+ 3236us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:39.912664 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:39.912789 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:39.914339 (+ 1550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:39.914346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:39.914375 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:39.914757 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.914770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.917241 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.917246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.920567 (+ 3321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:39.920586 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.923240 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.923250 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:39.924213 (+ 963us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.924244 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 20:37:39.924320 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 20:37:39.924645 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3051,"mutex_wait_us":34,"thread_start_us":2895,"threads_started":1}
I20260806 20:37:39.925534 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.909351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39600 (local address 127.23.200.2:33953)
0806 20:37:39.915162 (+ 5811us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:39.915167 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:39.915187 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:37:39.915262 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:39.915266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:39.915406 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:39.915595 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:39.916150 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.917897 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.921105 (+ 3208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.922766 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.924572 (+ 1806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.924745 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:39.924856 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 20:37:39.925058 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5683,"thread_start_us":5545,"threads_started":1}
I20260806 20:37:39.927223 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:39.910349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34678 (local address 127.23.200.3:35297)
0806 20:37:39.913641 (+ 3292us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:39.913649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:39.913684 (+ 35us) server_negotiation.cc:407] Connection header received
0806 20:37:39.913783 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:39.913792 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:39.914096 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:39.914355 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:39.914972 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.917088 (+ 2116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.920732 (+ 3644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:39.922820 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:39.925524 (+ 2704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:39.926469 (+ 945us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:39.926673 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 20:37:39.926875 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3100,"thread_start_us":2906,"threads_started":1}
I20260806 20:37:39.928349 25087 tablet_service.cc:1467] Processing CreateTablet for tablet c2102581e66e40479a03e92674022767 (DEFAULT_TABLE table=test-table [id=8f9f3488b3cc4c17baa72397cf725df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:39.930011 25087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2102581e66e40479a03e92674022767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:39.929538 25227 tablet_service.cc:1467] Processing CreateTablet for tablet c2102581e66e40479a03e92674022767 (DEFAULT_TABLE table=test-table [id=8f9f3488b3cc4c17baa72397cf725df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:39.931042 25227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2102581e66e40479a03e92674022767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:39.933507 25370 tablet_service.cc:1467] Processing CreateTablet for tablet c2102581e66e40479a03e92674022767 (DEFAULT_TABLE table=test-table [id=8f9f3488b3cc4c17baa72397cf725df9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:39.934767 25370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2102581e66e40479a03e92674022767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:39.948611 25458 tablet_bootstrap.cc:492] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: Bootstrap starting.
I20260806 20:37:39.949714 25459 tablet_bootstrap.cc:492] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: Bootstrap starting.
I20260806 20:37:39.951248 25460 tablet_bootstrap.cc:492] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: Bootstrap starting.
I20260806 20:37:39.952982 25458 tablet_bootstrap.cc:654] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:39.953039 25459 tablet_bootstrap.cc:654] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:39.954627 25458 log.cc:824] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:39.954845 25459 log.cc:824] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:39.955391 25460 tablet_bootstrap.cc:654] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:39.957295 25460 log.cc:824] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:39.959024 25459 tablet_bootstrap.cc:492] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: No bootstrap required, opened a new log
I20260806 20:37:39.959370 25458 tablet_bootstrap.cc:492] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: No bootstrap required, opened a new log
I20260806 20:37:39.959414 25459 ts_tablet_manager.cc:1397] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 20:37:39.959735 25458 ts_tablet_manager.cc:1397] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260806 20:37:39.961158 25460 tablet_bootstrap.cc:492] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: No bootstrap required, opened a new log
I20260806 20:37:39.961494 25460 ts_tablet_manager.cc:1397] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 20:37:39.969112 25458 raft_consensus.cc:348] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:39.969640 25458 raft_consensus.cc:374] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:39.969739 25458 raft_consensus.cc:729] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4e9493efff0475198b4dd45de736ce2, State: Initialized, Role: FOLLOWER
I20260806 20:37:39.970398 25458 consensus_queue.cc:260] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:39.971606 25438 heartbeater.cc:503] Master 127.23.200.62:38713 was elected leader, sending a full tablet report...
I20260806 20:37:39.972143 25458 ts_tablet_manager.cc:1428] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: Time spent starting tablet: real 0.012s user 0.007s sys 0.005s
I20260806 20:37:39.973047 25459 raft_consensus.cc:348] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:39.973563 25459 raft_consensus.cc:374] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:39.973737 25459 raft_consensus.cc:729] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2396d13124034f7e9f21f9708b6deeb8, State: Initialized, Role: FOLLOWER
I20260806 20:37:39.973845 25460 raft_consensus.cc:348] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:39.974452 25460 raft_consensus.cc:374] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:39.974373 25459 consensus_queue.cc:260] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:39.974601 25460 raft_consensus.cc:729] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21f76d5481e24cd4a14ce070c1e417bf, State: Initialized, Role: FOLLOWER
I20260806 20:37:39.975152 25460 consensus_queue.cc:260] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:39.978207 25459 ts_tablet_manager.cc:1428] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: Time spent starting tablet: real 0.019s user 0.007s sys 0.011s
I20260806 20:37:39.980096 25295 heartbeater.cc:503] Master 127.23.200.62:38713 was elected leader, sending a full tablet report...
I20260806 20:37:39.980720 25460 ts_tablet_manager.cc:1428] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: Time spent starting tablet: real 0.019s user 0.000s sys 0.020s
W20260806 20:37:40.068706 25296 tablet.cc:2369] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:37:40.072947 25439 tablet.cc:2369] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:37:40.075876 25156 tablet.cc:2369] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:37:40.314947 25465 raft_consensus.cc:482] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:40.315310 25465 raft_consensus.cc:504] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:40.317276 25465 leader_election.cc:288] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21f76d5481e24cd4a14ce070c1e417bf (127.23.200.2:33953), d4e9493efff0475198b4dd45de736ce2 (127.23.200.3:35297)
I20260806 20:37:40.327378 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.318215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:39727 (local address 127.23.200.3:35297)
0806 20:37:40.318406 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.318413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.319740 (+ 1327us) server_negotiation.cc:407] Connection header received
0806 20:37:40.320144 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.320150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.320236 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.320342 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:40.321311 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.322249 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.324885 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.326326 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.326534 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.326838 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.326898 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.327029 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 20:37:40.327865 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.317565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:35297 (local address 127.23.200.1:39727)
0806 20:37:40.319624 (+ 2059us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:40.319644 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:40.319913 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:40.320373 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:40.320378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:40.320401 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:40.321135 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:40.321150 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.322408 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.322416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:40.324706 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:40.324722 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.326553 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.326559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:40.326743 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.326778 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:37:40.326860 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 20:37:40.327029 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1824,"mutex_wait_us":25,"spinlock_wait_cycles":896,"thread_start_us":2164,"threads_started":1}
I20260806 20:37:40.327929 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.318480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:39329 (local address 127.23.200.2:33953)
0806 20:37:40.318759 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.318767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.320158 (+ 1391us) server_negotiation.cc:407] Connection header received
0806 20:37:40.320345 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.320349 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.320441 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.320540 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:40.321702 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.322868 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.325323 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.326627 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.326747 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.326810 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.327423 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.327587 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 20:37:40.328536 25467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.317688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33953 (local address 127.23.200.1:39329)
0806 20:37:40.320043 (+ 2355us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:40.320062 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:40.320193 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:40.320581 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:40.320587 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:40.320641 (+ 54us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:40.321478 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:40.321492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.323028 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.323034 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:40.325094 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:40.325104 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.327051 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.327058 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:40.327232 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.327261 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 20:37:40.328099 (+ 838us) client_negotiation.cc:240] Negotiation successful
0806 20:37:40.328234 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2196,"mutex_wait_us":20,"thread_start_us":1368,"threads_started":1}
I20260806 20:37:40.329342 25391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2102581e66e40479a03e92674022767" candidate_uuid: "2396d13124034f7e9f21f9708b6deeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e9493efff0475198b4dd45de736ce2" is_pre_election: true
I20260806 20:37:40.329824 25391 raft_consensus.cc:2454] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2396d13124034f7e9f21f9708b6deeb8 in term 0.
I20260806 20:37:40.329932 25248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2102581e66e40479a03e92674022767" candidate_uuid: "2396d13124034f7e9f21f9708b6deeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f76d5481e24cd4a14ce070c1e417bf" is_pre_election: true
I20260806 20:37:40.330518 25248 raft_consensus.cc:2454] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2396d13124034f7e9f21f9708b6deeb8 in term 0.
I20260806 20:37:40.330561 25041 leader_election.cc:302] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2396d13124034f7e9f21f9708b6deeb8, d4e9493efff0475198b4dd45de736ce2; no voters:
I20260806 20:37:40.331005 25465 raft_consensus.cc:2781] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:40.331198 25465 raft_consensus.cc:482] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:40.331318 25465 raft_consensus.cc:3037] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:40.333920 25465 raft_consensus.cc:504] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:40.334728 25465 leader_election.cc:288] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [CANDIDATE]: Term 1 election: Requested vote from peers 21f76d5481e24cd4a14ce070c1e417bf (127.23.200.2:33953), d4e9493efff0475198b4dd45de736ce2 (127.23.200.3:35297)
I20260806 20:37:40.335016 25248 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2102581e66e40479a03e92674022767" candidate_uuid: "2396d13124034f7e9f21f9708b6deeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21f76d5481e24cd4a14ce070c1e417bf"
I20260806 20:37:40.335135 25391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2102581e66e40479a03e92674022767" candidate_uuid: "2396d13124034f7e9f21f9708b6deeb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4e9493efff0475198b4dd45de736ce2"
I20260806 20:37:40.335212 25248 raft_consensus.cc:3037] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:40.335363 25391 raft_consensus.cc:3037] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:40.337494 25391 raft_consensus.cc:2454] T c2102581e66e40479a03e92674022767 P d4e9493efff0475198b4dd45de736ce2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2396d13124034f7e9f21f9708b6deeb8 in term 1.
I20260806 20:37:40.337688 25248 raft_consensus.cc:2454] T c2102581e66e40479a03e92674022767 P 21f76d5481e24cd4a14ce070c1e417bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2396d13124034f7e9f21f9708b6deeb8 in term 1.
I20260806 20:37:40.337913 25041 leader_election.cc:302] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2396d13124034f7e9f21f9708b6deeb8, d4e9493efff0475198b4dd45de736ce2; no voters:
I20260806 20:37:40.338243 25465 raft_consensus.cc:2781] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:40.338845 25465 raft_consensus.cc:686] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [term 1 LEADER]: Becoming Leader. State: Replica: 2396d13124034f7e9f21f9708b6deeb8, State: Running, Role: LEADER
I20260806 20:37:40.339512 25465 consensus_queue.cc:237] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } }
I20260806 20:37:40.344287 24954 catalog_manager.cc:5495] T c2102581e66e40479a03e92674022767 P 2396d13124034f7e9f21f9708b6deeb8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2396d13124034f7e9f21f9708b6deeb8 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "2396d13124034f7e9f21f9708b6deeb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21f76d5481e24cd4a14ce070c1e417bf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4e9493efff0475198b4dd45de736ce2" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2396d13124034f7e9f21f9708b6deeb8" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 46869 } health_report { overall_health: HEALTHY } } }
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:40.388978 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.368683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.23.200.1:46869)
0806 20:37:40.368880 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.368884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.369825 (+ 941us) server_negotiation.cc:407] Connection header received
0806 20:37:40.369949 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.369953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.370027 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.370110 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:40.372044 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.372928 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.374420 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.374920 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.375018 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.383409 (+ 8391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:40.383424 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:40.383441 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:40.383473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:40.386292 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.387015 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.387022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.387024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.387200 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.387553 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.387556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.387558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.387996 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:40.388294 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.388444 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.388673 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048660, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048660, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:40.436846 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.422746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44040 (local address 127.23.200.1:46869)
0806 20:37:40.422979 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.422984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.423134 (+ 150us) server_negotiation.cc:407] Connection header received
0806 20:37:40.423278 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.423282 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.423348 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.423430 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:40.424802 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.425532 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.426943 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.427391 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.427545 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.431815 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:40.431820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:40.431824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:40.431855 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:40.434667 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.435225 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.435229 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.435231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.435395 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.435734 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.435738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.435740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.436000 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:40.436216 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.436409 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.436581 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:40.481882 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.467476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44052 (local address 127.23.200.1:46869)
0806 20:37:40.467693 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.467701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.468046 (+ 345us) server_negotiation.cc:407] Connection header received
0806 20:37:40.468252 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.468256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.468326 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.468435 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:40.469902 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.470722 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.472050 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.472453 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.472596 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.476731 (+ 4135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:40.476736 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:40.476740 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:40.476770 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:40.479409 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.480069 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.480074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.480077 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.480247 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.480587 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.480591 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.480593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.480828 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:40.481020 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.481433 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.481617 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260806 20:37:40.484072 25065 tablet_service.cc:2908] Rejecting scan request for tablet c2102581e66e40479a03e92674022767: Uninitialized: clean time has not yet been initialized
I20260806 20:37:40.487013 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25020
I20260806 20:37:40.499748 25457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.488025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44054 (local address 127.23.200.1:46869)
0806 20:37:40.488181 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.488185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.488564 (+ 379us) server_negotiation.cc:407] Connection header received
0806 20:37:40.488686 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.488689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.488742 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.488817 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:40.489903 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.490616 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.492105 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.492486 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.492572 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.495065 (+ 2493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:40.495069 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:40.495073 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:40.495100 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:40.497649 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.498215 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.498219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.498221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.498378 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.498694 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.498697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.498699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.498921 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:40.499099 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.499359 (+ 260us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.499501 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 20:37:40.500412 25150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:40.591658 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25020
I20260806 20:37:40.611246 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25161
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:37:40.626428 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.612183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39614 (local address 127.23.200.2:33953)
0806 20:37:40.612379 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.612386 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.612408 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:37:40.612674 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.612679 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.612747 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.612874 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:40.614234 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.615058 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.616423 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.616812 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.616955 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.621216 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:40.621230 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:40.621247 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:40.621278 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:40.623979 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.624664 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.624672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.624674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.624807 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.625130 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.625133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.625135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.625500 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:40.625721 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.625955 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.626114 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:37:40.627138 25290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:40.714138 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25161
I20260806 20:37:40.732033 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25300
Aug 06 20:37:40 dist-test-slave-tm1g krb5kdc[24908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:37:40.747227 25455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:40.732819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34688 (local address 127.23.200.3:35297)
0806 20:37:40.733015 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:40.733023 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:40.733049 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:37:40.733265 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:40.733270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:40.733333 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:40.733444 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:40.734677 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.735397 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.736629 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:40.737048 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:40.737180 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:40.741583 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:40.741598 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:40.741614 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:40.741654 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:40.744394 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.745048 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.745056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.745059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.745191 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:40.745592 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:40.745598 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:40.745600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:40.745937 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:40.746244 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:40.746758 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 20:37:40.746962 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:37:40.747888 25433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:40.835004 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25300
I20260806 20:37:40.852582 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 24921
I20260806 20:37:40.853746 24987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:40.961781 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 24921
2026-08-06T20:37:40Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4423 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:37:41 dist-test-slave-tm1g krb5kdc[25489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:37:41 dist-test-slave-tm1g krb5kdc[25489](info): set up 2 sockets
Aug 06 20:37:41 dist-test-slave-tm1g krb5kdc[25489](info): commencing operation
krb5kdc: starting...
W20260806 20:37:42.061599 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:37:42 dist-test-slave-tm1g krb5kdc[25489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:37:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:37:42.239233 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35337
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:42.561239 25503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:42.561512 25503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:42.561689 25503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:42.568713 25503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:37:42.568813 25503 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 20:37:42.568895 25503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:42.568991 25503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:42.569089 25503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:37:42.569193 25503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:37:42.578816 25503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35337
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:42.580044 25503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:42.581357 25503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:42.593034 25508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:42.593075 25509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:42.593233 25511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:42.593837 25503 server_base.cc:1034] running on GCE node
I20260806 20:37:42.594653 25503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:42.595932 25503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:42.597180 25503 hybrid_clock.cc:648] HybridClock initialized: now 1786048662597123 us; error 34 us; skew 500 ppm
I20260806 20:37:42.597506 25503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:42 dist-test-slave-tm1g krb5kdc[25489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048662, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:42.600519 25503 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:37:42.601956 25503 webserver.cc:460] Webserver started at http://127.23.200.62:37291/ using document root <none> and password file <none>
I20260806 20:37:42.602728 25503 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:42.602859 25503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:42.603286 25503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:42.606154 25503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "717d1480f7db4cda978244b0451e2f65"
format_stamp: "Formatted at 2026-08-06 20:37:42 on dist-test-slave-tm1g"
server_key: "e6dceb37a75468c0f48b727fa513982e"
server_key_iv: "77d5e6f4970423117f33f69f68abc070"
server_key_version: "encryptionkey@0"
I20260806 20:37:42.606922 25503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "717d1480f7db4cda978244b0451e2f65"
format_stamp: "Formatted at 2026-08-06 20:37:42 on dist-test-slave-tm1g"
server_key: "e6dceb37a75468c0f48b727fa513982e"
server_key_iv: "77d5e6f4970423117f33f69f68abc070"
server_key_version: "encryptionkey@0"
I20260806 20:37:42.611835 25503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260806 20:37:42.615058 25518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:42.616349 25503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 20:37:42.616508 25503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "717d1480f7db4cda978244b0451e2f65"
format_stamp: "Formatted at 2026-08-06 20:37:42 on dist-test-slave-tm1g"
server_key: "e6dceb37a75468c0f48b727fa513982e"
server_key_iv: "77d5e6f4970423117f33f69f68abc070"
server_key_version: "encryptionkey@0"
I20260806 20:37:42.616677 25503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:42.644250 25503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:42.648257 25503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:42.648476 25503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:42.670233 25503 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:35337
I20260806 20:37:42.670254 25581 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:35337 every 8 connection(s)
I20260806 20:37:42.671572 25503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:37:42.674556 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25503
I20260806 20:37:42.674770 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:37:42.675251 25582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:42.675455 24352 external_mini_cluster.cc:1428] Setting key ccf6c11d8d7e42eadea158558f39b204
I20260806 20:37:42.689548 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: Bootstrap starting.
Aug 06 20:37:42 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:42.693292 25582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:42.694416 25582 log.cc:824] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:42.697572 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: No bootstrap required, opened a new log
I20260806 20:37:42.698601 25585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:42.677395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38086 (local address 127.23.200.62:35337)
0806 20:37:42.678288 (+ 893us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:42.678329 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:42.678378 (+ 49us) server_negotiation.cc:407] Connection header received
0806 20:37:42.680219 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:42.680248 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:42.680745 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:42.681144 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:42.682438 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:42.684423 (+ 1985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:42.685761 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:42.686878 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:42.687076 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:42.691390 (+ 4314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:42.691397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:42.691410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:42.691449 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:42.694530 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:42.695328 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:42.695339 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:42.695349 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:42.695587 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:42.696026 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:42.696029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:42.696031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:42.696502 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:42.696726 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:42.697200 (+ 474us) server_negotiation.cc:299] Negotiation successful
0806 20:37:42.697526 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":374,"threads_started":1}
I20260806 20:37:42.705412 25582 raft_consensus.cc:348] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717d1480f7db4cda978244b0451e2f65" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35337 } }
I20260806 20:37:42.705755 25582 raft_consensus.cc:374] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:42.705866 25582 raft_consensus.cc:729] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717d1480f7db4cda978244b0451e2f65, State: Initialized, Role: FOLLOWER
I20260806 20:37:42.706513 25582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717d1480f7db4cda978244b0451e2f65" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35337 } }
I20260806 20:37:42.706774 25582 raft_consensus.cc:388] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:37:42.706915 25582 raft_consensus.cc:482] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:37:42.707098 25582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:42.709122 25582 raft_consensus.cc:504] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717d1480f7db4cda978244b0451e2f65" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35337 } }
I20260806 20:37:42.709601 25582 leader_election.cc:302] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 717d1480f7db4cda978244b0451e2f65; no voters:
I20260806 20:37:42.710243 25582 leader_election.cc:288] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:37:42.710398 25587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:42.711231 25587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [term 1 LEADER]: Becoming Leader. State: Replica: 717d1480f7db4cda978244b0451e2f65, State: Running, Role: LEADER
I20260806 20:37:42.711804 25587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717d1480f7db4cda978244b0451e2f65" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35337 } }
I20260806 20:37:42.712488 25582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:37:42.714668 25588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "717d1480f7db4cda978244b0451e2f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717d1480f7db4cda978244b0451e2f65" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35337 } } }
I20260806 20:37:42.715029 25588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:42.718423 25593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:37:42.718400 25589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 717d1480f7db4cda978244b0451e2f65. Latest consensus state: current_term: 1 leader_uuid: "717d1480f7db4cda978244b0451e2f65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "717d1480f7db4cda978244b0451e2f65" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35337 } } }
I20260806 20:37:42.718729 25589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65 [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:42.724583 25593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:37:42.733912 25593 catalog_manager.cc:1347] Generated new cluster ID: 1afd55187fd94d8f8fb55de6727932d5
I20260806 20:37:42.734031 25593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:37:42.752977 25593 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:37:42.754209 25593 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:37:42.764858 25593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: Generated new TSK 0
I20260806 20:37:42.765664 25593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:37:42.816248 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35337
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:43.135617 25609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:43.135908 25609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:43.136094 25609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:43.143019 25609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:43.143159 25609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:43.143324 25609 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:37:43.143461 25609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:37:43.152661 25609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:35337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:43.153893 25609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:43.155229 25609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:43.168457 25615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:43.168497 25616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:43.168411 25618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:43.169391 25609 server_base.cc:1034] running on GCE node
I20260806 20:37:43.169937 25609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:43.170852 25609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:43.172065 25609 hybrid_clock.cc:648] HybridClock initialized: now 1786048663172003 us; error 34 us; skew 500 ppm
I20260806 20:37:43.172351 25609 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:43 dist-test-slave-tm1g krb5kdc[25489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048663, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:43.175355 25609 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:37:43.176789 25609 webserver.cc:460] Webserver started at http://127.23.200.1:36337/ using document root <none> and password file <none>
I20260806 20:37:43.177470 25609 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:43.177611 25609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:43.177940 25609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:43.180917 25609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "c7fac236ac204a06b96b662c83708eed"
format_stamp: "Formatted at 2026-08-06 20:37:43 on dist-test-slave-tm1g"
server_key: "62448a87dc841a2b6f8eab5ba4f82cab"
server_key_iv: "a387503fda958f1d7f1c43de14901a8f"
server_key_version: "encryptionkey@0"
I20260806 20:37:43.181715 25609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "c7fac236ac204a06b96b662c83708eed"
format_stamp: "Formatted at 2026-08-06 20:37:43 on dist-test-slave-tm1g"
server_key: "62448a87dc841a2b6f8eab5ba4f82cab"
server_key_iv: "a387503fda958f1d7f1c43de14901a8f"
server_key_version: "encryptionkey@0"
I20260806 20:37:43.186776 25609 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260806 20:37:43.190220 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:43.191468 25609 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:43.191650 25609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "c7fac236ac204a06b96b662c83708eed"
format_stamp: "Formatted at 2026-08-06 20:37:43 on dist-test-slave-tm1g"
server_key: "62448a87dc841a2b6f8eab5ba4f82cab"
server_key_iv: "a387503fda958f1d7f1c43de14901a8f"
server_key_version: "encryptionkey@0"
I20260806 20:37:43.191864 25609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:43.214640 25609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:43.218699 25609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:43.218962 25609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:43.221808 25609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:43.221921 25609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:43.222123 25609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:43.222235 25609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 06 20:37:43 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048663, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:43.246446 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.222262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49393 (local address 127.23.200.62:35337)
0806 20:37:43.222733 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:43.222739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:43.224030 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 20:37:43.227186 (+ 3156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:43.227193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:43.227283 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:43.227401 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:43.229763 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.231064 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.234264 (+ 3200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.234798 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.234944 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.239688 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:43.239693 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:43.239696 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:43.239728 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:43.242367 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.243499 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.243507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.243510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.243675 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.244171 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.244179 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.244182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.244582 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:43.244728 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:43.245549 (+ 821us) server_negotiation.cc:299] Negotiation successful
0806 20:37:43.245861 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":225,"threads_started":1}
I20260806 20:37:43.246994 25636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.222973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.23.200.1:49393)
0806 20:37:43.223844 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:43.223900 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:43.226681 (+ 2781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:43.227826 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:43.227841 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:43.228677 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:43.229492 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.229569 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.231288 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.231294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.234087 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.234111 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.235098 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.235108 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.235491 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.236393 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:43.236431 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:43.239458 (+ 3027us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:43.242511 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.242520 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.242535 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.243288 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.243827 (+ 539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.243832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.243836 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.244044 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.244813 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:43.244819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:43.245143 (+ 324us) client_negotiation.cc:769] Sending connection context
0806 20:37:43.245385 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 20:37:43.245754 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":340,"threads_started":1}
I20260806 20:37:43.269805 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.252662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.23.200.1:42875)
0806 20:37:43.253727 (+ 1065us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:43.253755 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:43.253924 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:43.254878 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:43.254886 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:43.255698 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:43.256255 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.256273 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.257502 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.257509 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.259609 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.259630 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.260603 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.260612 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.260907 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.261595 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:43.261626 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:43.262948 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:43.267332 (+ 4384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.267336 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.267339 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.267725 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.268180 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.268184 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.268186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.268301 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.268992 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:43.268995 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:43.269154 (+ 159us) client_negotiation.cc:769] Sending connection context
0806 20:37:43.269257 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 20:37:43.269480 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":715,"threads_started":1}
I20260806 20:37:43.270215 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.252669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:42875 (local address 127.23.200.62:35337)
0806 20:37:43.252880 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:43.252888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:43.253864 (+ 976us) server_negotiation.cc:407] Connection header received
0806 20:37:43.254063 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:43.254066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:43.254124 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:43.254374 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:43.256427 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.257329 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.259791 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.260357 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.260504 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.263152 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:43.263157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:43.263160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:43.263189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:43.267194 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.267857 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.267865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.267871 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.268064 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.268417 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.268425 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.268431 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.268869 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:43.269059 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:43.269441 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 20:37:43.269734 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:37:43.283798 25609 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:39317
I20260806 20:37:43.283824 25753 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:39317 every 8 connection(s)
I20260806 20:37:43.284977 25609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:37:43.287060 25754 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35337
I20260806 20:37:43.287282 25754 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:43.287892 25754 heartbeater.cc:511] Master 127.23.200.62:35337 requested a full tablet report, sending...
I20260806 20:37:43.289740 25535 ts_manager.cc:194] Registered new tserver with Master: c7fac236ac204a06b96b662c83708eed (127.23.200.1:39317)
I20260806 20:37:43.291229 25535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:49393
I20260806 20:37:43.294845 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25609
I20260806 20:37:43.295081 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:37:43.295737 24352 external_mini_cluster.cc:1428] Setting key 486ea0adf6ae300145a481718ed20681
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:37:43.335760 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35337
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:43.651057 25759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:43.651360 25759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:43.651546 25759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:43.658529 25759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:43.658676 25759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:43.658830 25759 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:37:43.658977 25759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:37:43.668406 25759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:35337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:43.669755 25759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:43.671135 25759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:43.684868 25764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:43.685024 25765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:43.685355 25767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:43.686913 25759 server_base.cc:1034] running on GCE node
I20260806 20:37:43.687549 25759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:43.688467 25759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:43.689707 25759 hybrid_clock.cc:648] HybridClock initialized: now 1786048663689659 us; error 62 us; skew 500 ppm
I20260806 20:37:43.690033 25759 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:43 dist-test-slave-tm1g krb5kdc[25489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048663, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:43.693111 25759 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:37:43.694589 25759 webserver.cc:460] Webserver started at http://127.23.200.2:34809/ using document root <none> and password file <none>
I20260806 20:37:43.695355 25759 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:43.695500 25759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:43.695816 25759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:43.698772 25759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "b91c0f1653d2437c82231114723feed5"
format_stamp: "Formatted at 2026-08-06 20:37:43 on dist-test-slave-tm1g"
server_key: "1248b17e795b6ca4b17d363190c1a5ae"
server_key_iv: "99976893f0e26a5ffd9e94a5548b51c0"
server_key_version: "encryptionkey@0"
I20260806 20:37:43.699613 25759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "b91c0f1653d2437c82231114723feed5"
format_stamp: "Formatted at 2026-08-06 20:37:43 on dist-test-slave-tm1g"
server_key: "1248b17e795b6ca4b17d363190c1a5ae"
server_key_iv: "99976893f0e26a5ffd9e94a5548b51c0"
server_key_version: "encryptionkey@0"
I20260806 20:37:43.704550 25759 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:37:43.707835 25775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:43.709103 25759 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:37:43.709270 25759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "b91c0f1653d2437c82231114723feed5"
format_stamp: "Formatted at 2026-08-06 20:37:43 on dist-test-slave-tm1g"
server_key: "1248b17e795b6ca4b17d363190c1a5ae"
server_key_iv: "99976893f0e26a5ffd9e94a5548b51c0"
server_key_version: "encryptionkey@0"
I20260806 20:37:43.709486 25759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:43.738668 25759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:43.743852 25759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:43.744146 25759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:43.747443 25759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:43.747543 25759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:43.747711 25759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:43.747790 25759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:37:43 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048663, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:43.769009 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.748006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:37747 (local address 127.23.200.62:35337)
0806 20:37:43.748246 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:43.748254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:43.749776 (+ 1522us) server_negotiation.cc:407] Connection header received
0806 20:37:43.752283 (+ 2507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:43.752289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:43.752382 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:43.752496 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:43.753905 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.754774 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.758182 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.758839 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.758973 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.762459 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:43.762463 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:43.762466 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:43.762498 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:43.765239 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.766248 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.766256 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.766259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.766438 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.766992 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.767000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.767003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.767381 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:43.767553 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:43.768588 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0806 20:37:43.768757 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 20:37:43.771019 25784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.748682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.23.200.2:37747)
0806 20:37:43.749596 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:43.749652 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:43.751954 (+ 2302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:43.752718 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:43.752726 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:43.753191 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:43.753729 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.753746 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.755000 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.755009 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.757985 (+ 2976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.758023 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.759089 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.759097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.759346 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.759985 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:43.760007 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:43.762182 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:43.765428 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.765437 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.765451 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.766118 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.766563 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.766568 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.766571 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.766880 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.767577 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:43.767589 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:43.768131 (+ 542us) client_negotiation.cc:769] Sending connection context
0806 20:37:43.768510 (+ 379us) client_negotiation.cc:240] Negotiation successful
0806 20:37:43.769188 (+ 678us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":420,"threads_started":1}
I20260806 20:37:43.795389 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.777498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:45155 (local address 127.23.200.62:35337)
0806 20:37:43.777673 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:43.777682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:43.778694 (+ 1012us) server_negotiation.cc:407] Connection header received
0806 20:37:43.778878 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:43.778884 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:43.778981 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:43.779090 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:43.780680 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.781446 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.784024 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.784762 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.784908 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.787994 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:43.788001 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:43.788007 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:43.788051 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:43.791891 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.792966 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.792970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.792972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.793097 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:43.793525 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:43.793528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:43.793530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:43.793912 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:43.794083 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:43.794715 (+ 632us) server_negotiation.cc:299] Negotiation successful
0806 20:37:43.794970 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 20:37:43.796263 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:43.777690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.23.200.2:45155)
0806 20:37:43.778567 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:43.778590 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:43.778747 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:43.779155 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:43.779163 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:43.779920 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:43.780507 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.780526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.781597 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.781606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.783843 (+ 2237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:43.783864 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:43.785014 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:43.785028 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:43.785338 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:43.786199 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:43.786235 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:43.787728 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:43.792057 (+ 4329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.792067 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.792073 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.792829 (+ 756us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.793225 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:43.793232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:43.793238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:43.793407 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:43.794090 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:43.794099 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:43.794390 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 20:37:43.794586 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 20:37:43.795213 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":468,"threads_started":1}
I20260806 20:37:43.802110 25759 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:34903
I20260806 20:37:43.802148 25899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:34903 every 8 connection(s)
I20260806 20:37:43.803323 25759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:37:43.805334 25900 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35337
I20260806 20:37:43.805550 25900 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:43.806190 25900 heartbeater.cc:511] Master 127.23.200.62:35337 requested a full tablet report, sending...
I20260806 20:37:43.807556 25535 ts_manager.cc:194] Registered new tserver with Master: b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903)
I20260806 20:37:43.808298 25535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:37747
I20260806 20:37:43.812240 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25759
I20260806 20:37:43.812425 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:37:43.812966 24352 external_mini_cluster.cc:1428] Setting key 38629b545371468e9b571c1bbaeb8f84
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:37:43.852887 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35337
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:44.171151 25905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:44.171398 25905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:44.171501 25905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:44.178421 25905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:44.178519 25905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:44.178598 25905 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:37:44.178680 25905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:37:44.187834 25905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:35337
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:44.188962 25905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:44.190224 25905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:44.203436 25910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:44.203397 25913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:44.203471 25911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:44.204286 25905 server_base.cc:1034] running on GCE node
I20260806 20:37:44.204913 25905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:44.205807 25905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:44.207077 25905 hybrid_clock.cc:648] HybridClock initialized: now 1786048664207014 us; error 25 us; skew 500 ppm
I20260806 20:37:44.207432 25905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:44 dist-test-slave-tm1g krb5kdc[25489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:44.210564 25905 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:37:44.211851 25905 webserver.cc:460] Webserver started at http://127.23.200.3:41713/ using document root <none> and password file <none>
I20260806 20:37:44.212646 25905 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:44.212747 25905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:44.213071 25905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:44.216051 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "5588a0d804344ec6aeced87f56165107"
format_stamp: "Formatted at 2026-08-06 20:37:44 on dist-test-slave-tm1g"
server_key: "38c50a6580d54cb9f0cfac5ecc178b8b"
server_key_iv: "152452bc10716452d97b987f49d445f2"
server_key_version: "encryptionkey@0"
I20260806 20:37:44.216785 25905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "5588a0d804344ec6aeced87f56165107"
format_stamp: "Formatted at 2026-08-06 20:37:44 on dist-test-slave-tm1g"
server_key: "38c50a6580d54cb9f0cfac5ecc178b8b"
server_key_iv: "152452bc10716452d97b987f49d445f2"
server_key_version: "encryptionkey@0"
I20260806 20:37:44.222110 25905 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:37:44.225484 25922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:44.226842 25905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:44.226974 25905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "5588a0d804344ec6aeced87f56165107"
format_stamp: "Formatted at 2026-08-06 20:37:44 on dist-test-slave-tm1g"
server_key: "38c50a6580d54cb9f0cfac5ecc178b8b"
server_key_iv: "152452bc10716452d97b987f49d445f2"
server_key_version: "encryptionkey@0"
I20260806 20:37:44.227149 25905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:44.253197 25905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:44.256942 25905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:44.257174 25905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:44.259887 25905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:44.259979 25905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:44.260147 25905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:44.260244 25905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:37:44 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:44.284035 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.260253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:42569 (local address 127.23.200.62:35337)
0806 20:37:44.260499 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.260506 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.261851 (+ 1345us) server_negotiation.cc:407] Connection header received
0806 20:37:44.263944 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.263950 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.264044 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.264157 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:44.266157 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.267568 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.270067 (+ 2499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.270668 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.270831 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.276124 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:44.276133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:44.276139 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:44.276188 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:44.279572 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:44.280812 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:44.280817 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:44.280821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:44.281009 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:44.281559 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:44.281567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:44.281571 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:44.281976 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:44.282129 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.283150 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.283483 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 20:37:44.284693 25931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.260806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.23.200.3:42569)
0806 20:37:44.261665 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.261712 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.263653 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.264395 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.264402 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.265103 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:44.265919 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.265958 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.267776 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.267788 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.269917 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.269929 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.271048 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.271061 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.271411 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.272172 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:44.272209 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:44.275858 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:44.279727 (+ 3869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:44.279740 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:44.279766 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:44.280615 (+ 849us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:44.281163 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:44.281168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:44.281172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:44.281428 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:44.282122 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:44.282129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:44.282674 (+ 545us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.283008 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.283564 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":376,"threads_started":1}
I20260806 20:37:44.293776 25754 heartbeater.cc:503] Master 127.23.200.62:35337 was elected leader, sending a full tablet report...
I20260806 20:37:44.307602 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.292915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.23.200.3:51143)
0806 20:37:44.293515 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.293534 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.293720 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.294078 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.294083 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.294657 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:44.294999 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.295009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.296253 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.296261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.297463 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.297475 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.298340 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.298347 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.298535 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.299230 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:44.299261 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:44.300385 (+ 1124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:44.304220 (+ 3835us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:44.304230 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:44.304234 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:44.304713 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:44.305203 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:44.305208 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:44.305212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:44.305355 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:44.305995 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:44.305999 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:44.306385 (+ 386us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.306597 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.306917 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":599,"threads_started":1}
I20260806 20:37:44.308195 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.293321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:51143 (local address 127.23.200.62:35337)
0806 20:37:44.293501 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.293506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.293613 (+ 107us) server_negotiation.cc:407] Connection header received
0806 20:37:44.293839 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.293844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.293961 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.294050 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:44.295170 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.296104 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.297612 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.298130 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.298273 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.300641 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:44.300649 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:44.300654 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:44.300699 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:44.304031 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:44.304877 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:44.304883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:44.304886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:44.305093 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:44.305479 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:44.305487 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:44.305490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:44.305875 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:44.306042 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.307658 (+ 1616us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.307879 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:37:44.318538 25905 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:32821
I20260806 20:37:44.318568 26048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:32821 every 8 connection(s)
I20260806 20:37:44.319939 25905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:37:44.321938 26050 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35337
I20260806 20:37:44.322162 26050 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:44.322840 26050 heartbeater.cc:511] Master 127.23.200.62:35337 requested a full tablet report, sending...
I20260806 20:37:44.324193 25535 ts_manager.cc:194] Registered new tserver with Master: 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821)
I20260806 20:37:44.325028 25535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:42569
I20260806 20:37:44.328680 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 25905
I20260806 20:37:44.328873 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:37:44.329420 24352 external_mini_cluster.cc:1428] Setting key 12ef204faaff6693dae58674e63da1a1
I20260806 20:37:44.332425 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 20:37:44 dist-test-slave-tm1g krb5kdc[25489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 20:37:44 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:44.379050 25634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.364958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38102 (local address 127.23.200.62:35337)
0806 20:37:44.365185 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.365192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.365365 (+ 173us) server_negotiation.cc:407] Connection header received
0806 20:37:44.365587 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.365593 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.365668 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.365779 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:44.367251 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.368009 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.369281 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.369717 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.369858 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.373893 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:44.373898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:44.373901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:44.373933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:44.376430 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:44.377033 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:44.377039 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:44.377043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:44.377176 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:44.377554 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:44.377560 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:44.377563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:44.377859 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:44.378117 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.378447 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.378672 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:37:44.383044 25535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:37:44.390139 25535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:44.416887 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.402801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37704 (local address 127.23.200.1:39317)
0806 20:37:44.408787 (+ 5986us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.408794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.408826 (+ 32us) server_negotiation.cc:407] Connection header received
0806 20:37:44.408941 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.408953 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.409264 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.410077 (+ 813us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:44.410322 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.411691 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.414422 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.415733 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.415886 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.416116 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.416409 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.416578 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5810,"thread_start_us":5686,"threads_started":1}
I20260806 20:37:44.417057 26063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.402286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39317 (local address 127.0.0.1:37704)
0806 20:37:44.403500 (+ 1214us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.403563 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.403787 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.409528 (+ 5741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.409535 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.409564 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:44.410077 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.410094 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.411839 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.411847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.414170 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.414182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.415984 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.415991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.416195 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.416233 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.416365 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.416554 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"mutex_wait_us":24,"thread_start_us":718,"threads_started":1}
I20260806 20:37:44.418089 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.402286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:32821 (local address 127.0.0.1:56654)
0806 20:37:44.408141 (+ 5855us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.408160 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.408274 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.409967 (+ 1693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.409979 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.410015 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:44.410507 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.410522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.412780 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.412787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.415175 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.415190 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.417219 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.417230 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.417395 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.417433 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.417504 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.417642 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5690,"mutex_wait_us":72,"thread_start_us":5753,"threads_started":1}
I20260806 20:37:44.418943 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.409538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:34903 (local address 127.0.0.1:35168)
0806 20:37:44.410198 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.410215 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.410455 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.411329 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.411351 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.411380 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:44.411751 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.411762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.413719 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.413725 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.415980 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.415995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.418050 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.418058 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.418313 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.418354 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.418447 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.418575 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":1,"thread_start_us":304,"threads_started":1}
I20260806 20:37:44.419274 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.408678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35168 (local address 127.23.200.2:34903)
0806 20:37:44.409528 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.409536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.410341 (+ 805us) server_negotiation.cc:407] Connection header received
0806 20:37:44.410658 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.410665 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.410901 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.411076 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:44.411944 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.413534 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.416169 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.417518 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.417656 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.417849 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.418496 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.418671 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":571,"threads_started":1}
I20260806 20:37:44.419421 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.402520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.23.200.3:32821)
0806 20:37:44.409320 (+ 6800us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.409331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.409364 (+ 33us) server_negotiation.cc:407] Connection header received
0806 20:37:44.409458 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.409465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.409722 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.410695 (+ 973us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:44.411043 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.412599 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.415358 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.416766 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.418480 (+ 1714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.418706 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.418773 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.418941 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6610,"thread_start_us":6441,"threads_started":1}
I20260806 20:37:44.421253 25680 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9a835a53c74b65945274c81b05bcff (DEFAULT_TABLE table=test-table [id=1dc4d8f1bc5e4a22a7a5b69c157cf561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:44.422837 25680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9a835a53c74b65945274c81b05bcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:44.422844 25973 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9a835a53c74b65945274c81b05bcff (DEFAULT_TABLE table=test-table [id=1dc4d8f1bc5e4a22a7a5b69c157cf561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:44.422878 25832 tablet_service.cc:1467] Processing CreateTablet for tablet 3f9a835a53c74b65945274c81b05bcff (DEFAULT_TABLE table=test-table [id=1dc4d8f1bc5e4a22a7a5b69c157cf561]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:44.423995 25832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9a835a53c74b65945274c81b05bcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:44.424149 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f9a835a53c74b65945274c81b05bcff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:44.435134 26069 tablet_bootstrap.cc:492] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Bootstrap starting.
I20260806 20:37:44.437242 26070 tablet_bootstrap.cc:492] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Bootstrap starting.
I20260806 20:37:44.438853 26069 tablet_bootstrap.cc:654] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:44.439009 26071 tablet_bootstrap.cc:492] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Bootstrap starting.
I20260806 20:37:44.440575 26069 log.cc:824] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:44.441284 26070 tablet_bootstrap.cc:654] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:44.442314 26071 tablet_bootstrap.cc:654] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:44.442692 26070 log.cc:824] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:44.443914 26071 log.cc:824] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:44.444293 26069 tablet_bootstrap.cc:492] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: No bootstrap required, opened a new log
I20260806 20:37:44.444579 26069 ts_tablet_manager.cc:1397] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260806 20:37:44.445870 26070 tablet_bootstrap.cc:492] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: No bootstrap required, opened a new log
I20260806 20:37:44.446153 26070 ts_tablet_manager.cc:1397] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 20:37:44.446925 26071 tablet_bootstrap.cc:492] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: No bootstrap required, opened a new log
I20260806 20:37:44.447170 26071 ts_tablet_manager.cc:1397] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 20:37:44.454890 26071 raft_consensus.cc:348] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.455257 26071 raft_consensus.cc:374] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:44.455389 26071 raft_consensus.cc:729] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5588a0d804344ec6aeced87f56165107, State: Initialized, Role: FOLLOWER
I20260806 20:37:44.456017 26071 consensus_queue.cc:260] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.456838 26069 raft_consensus.cc:348] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.457378 26069 raft_consensus.cc:374] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:44.457438 26050 heartbeater.cc:503] Master 127.23.200.62:35337 was elected leader, sending a full tablet report...
I20260806 20:37:44.457566 26069 raft_consensus.cc:729] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91c0f1653d2437c82231114723feed5, State: Initialized, Role: FOLLOWER
I20260806 20:37:44.458176 26071 ts_tablet_manager.cc:1428] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 20:37:44.458153 26069 consensus_queue.cc:260] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.459503 25900 heartbeater.cc:503] Master 127.23.200.62:35337 was elected leader, sending a full tablet report...
I20260806 20:37:44.459348 26070 raft_consensus.cc:348] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.460093 26070 raft_consensus.cc:374] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:44.460162 26069 ts_tablet_manager.cc:1428] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Time spent starting tablet: real 0.015s user 0.009s sys 0.006s
I20260806 20:37:44.460294 26070 raft_consensus.cc:729] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fac236ac204a06b96b662c83708eed, State: Initialized, Role: FOLLOWER
I20260806 20:37:44.460999 26070 consensus_queue.cc:260] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.463449 26070 ts_tablet_manager.cc:1428] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Time spent starting tablet: real 0.017s user 0.016s sys 0.001s
W20260806 20:37:44.537518 25755 tablet.cc:2369] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:37:44.555378 25901 tablet.cc:2369] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:37:44.571568 26051 tablet.cc:2369] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:37:44.687853 26077 raft_consensus.cc:482] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:44.688217 26077 raft_consensus.cc:504] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.690037 26077 leader_election.cc:288] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821), b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903)
I20260806 20:37:44.698257 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.690522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:58861 (local address 127.23.200.3:32821)
0806 20:37:44.690793 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.690802 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.691456 (+ 654us) server_negotiation.cc:407] Connection header received
0806 20:37:44.691694 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.691701 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.691819 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.691922 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:44.692871 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.693883 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.696497 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.697315 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.697513 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.697599 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.697869 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.697998 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 20:37:44.698523 26078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.690455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:32821 (local address 127.23.200.1:58861)
0806 20:37:44.691328 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.691344 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.691530 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.692077 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.692081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.692113 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:44.692705 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.692721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.694009 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.694017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.696347 (+ 2330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.696359 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.697529 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.697535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.697708 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.697732 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.697842 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.697972 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":740,"spinlock_wait_cycles":16640,"thread_start_us":586,"threads_started":1}
I20260806 20:37:44.698843 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.691882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:41053 (local address 127.23.200.2:34903)
0806 20:37:44.692107 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.692113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.692350 (+ 237us) server_negotiation.cc:407] Connection header received
0806 20:37:44.692511 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.692519 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.692620 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.692728 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:44.693401 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.694393 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.696827 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.697811 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.698020 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.698089 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.698391 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.698516 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 20:37:44.699110 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.691451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:34903 (local address 127.23.200.1:41053)
0806 20:37:44.692248 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:44.692265 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:44.692380 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:44.692743 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:44.692746 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:44.692763 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:44.693217 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.693232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.694530 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.694535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.696678 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:44.696690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.698037 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.698043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:44.698213 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.698237 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 20:37:44.698365 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 20:37:44.698505 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":8,"spinlock_wait_cycles":1280,"thread_start_us":401,"threads_started":1}
I20260806 20:37:44.700062 25997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9a835a53c74b65945274c81b05bcff" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5588a0d804344ec6aeced87f56165107" is_pre_election: true
I20260806 20:37:44.700063 25854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9a835a53c74b65945274c81b05bcff" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91c0f1653d2437c82231114723feed5" is_pre_election: true
I20260806 20:37:44.700652 25997 raft_consensus.cc:2454] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 0.
I20260806 20:37:44.700691 25854 raft_consensus.cc:2454] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 0.
I20260806 20:37:44.701345 25630 leader_election.cc:302] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5588a0d804344ec6aeced87f56165107, c7fac236ac204a06b96b662c83708eed; no voters:
I20260806 20:37:44.701718 26077 raft_consensus.cc:2781] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:44.701872 26077 raft_consensus.cc:482] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:44.702004 26077 raft_consensus.cc:3037] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:44.704576 26077 raft_consensus.cc:504] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.705171 26077 leader_election.cc:288] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 election: Requested vote from peers 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821), b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903)
I20260806 20:37:44.705658 25997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9a835a53c74b65945274c81b05bcff" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5588a0d804344ec6aeced87f56165107"
I20260806 20:37:44.705863 25997 raft_consensus.cc:3037] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:44.705812 25854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f9a835a53c74b65945274c81b05bcff" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91c0f1653d2437c82231114723feed5"
I20260806 20:37:44.706070 25854 raft_consensus.cc:3037] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:44.708652 25854 raft_consensus.cc:2454] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 1.
I20260806 20:37:44.708685 25997 raft_consensus.cc:2454] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 1.
I20260806 20:37:44.709084 25630 leader_election.cc:302] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b91c0f1653d2437c82231114723feed5, c7fac236ac204a06b96b662c83708eed; no voters:
I20260806 20:37:44.709390 26077 raft_consensus.cc:2781] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:44.710058 26077 raft_consensus.cc:686] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 1 LEADER]: Becoming Leader. State: Replica: c7fac236ac204a06b96b662c83708eed, State: Running, Role: LEADER
I20260806 20:37:44.710737 26077 consensus_queue.cc:237] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } }
I20260806 20:37:44.715673 25535 catalog_manager.cc:5495] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed reported cstate change: term changed from 0 to 1, leader changed from <none> to c7fac236ac204a06b96b662c83708eed (127.23.200.1). New cstate: current_term: 1 leader_uuid: "c7fac236ac204a06b96b662c83708eed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } health_report { overall_health: UNKNOWN } } }
I20260806 20:37:44.783962 26066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:44.778473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37712 (local address 127.23.200.1:39317)
0806 20:37:44.778716 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:44.778721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:44.778814 (+ 93us) server_negotiation.cc:407] Connection header received
0806 20:37:44.779107 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:44.779112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:44.779191 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:44.779307 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:44.779677 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.780382 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.781749 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:44.782243 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:44.782415 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:44.782853 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:44.783359 (+ 506us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:44.783463 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:44.783587 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 20:37:44.783650 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 20:37:44.793148 25997 raft_consensus.cc:1261] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Refusing update from remote peer c7fac236ac204a06b96b662c83708eed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:44.793215 25854 raft_consensus.cc:1261] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Refusing update from remote peer c7fac236ac204a06b96b662c83708eed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:44.794346 26081 consensus_queue.cc:1035] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:44.794847 26077 consensus_queue.cc:1035] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:44.807503 26084 mvcc.cc:205] Tried to move back new op lower bound from 7315655330980761600 to 7315655330658656256. Current Snapshot: MvccSnapshot[applied={T|T < 7315655330980761600}]
I20260806 20:37:44.834254 25535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:44.834641 25535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:44.840102 25535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: Sending DeleteTablet for 3 replicas of tablet 3f9a835a53c74b65945274c81b05bcff
I20260806 20:37:44.841315 25973 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9a835a53c74b65945274c81b05bcff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:44 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56654
I20260806 20:37:44.841533 25680 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9a835a53c74b65945274c81b05bcff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:44 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:37704
I20260806 20:37:44.842185 25832 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f9a835a53c74b65945274c81b05bcff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:44 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35168
I20260806 20:37:44.842358 26095 tablet_replica.cc:330] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: stopping tablet replica
I20260806 20:37:44.843372 25535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:37:44.844023 25535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:44.844903 26095 raft_consensus.cc:2229] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:44.845386 26095 raft_consensus.cc:2258] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:44.848413 26097 tablet_replica.cc:330] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: stopping tablet replica
I20260806 20:37:44.849157 26097 raft_consensus.cc:2229] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:44.849906 26097 raft_consensus.cc:2258] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:44.851439 26095 ts_tablet_manager.cc:1905] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:44.852156 25973 tablet_service.cc:1467] Processing CreateTablet for tablet bdaa7f004db54982a836a07f34d7e461 (DEFAULT_TABLE table=test-table [id=84d16b65220a4714bdd5170fcec4c0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:44.852293 26097 ts_tablet_manager.cc:1905] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:44.852604 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdaa7f004db54982a836a07f34d7e461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:44.854713 25832 tablet_service.cc:1467] Processing CreateTablet for tablet bdaa7f004db54982a836a07f34d7e461 (DEFAULT_TABLE table=test-table [id=84d16b65220a4714bdd5170fcec4c0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:44.855244 25832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdaa7f004db54982a836a07f34d7e461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:44.859972 26071 tablet_bootstrap.cc:492] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Bootstrap starting.
I20260806 20:37:44.862246 26071 tablet_bootstrap.cc:654] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:44.862255 26097 ts_tablet_manager.cc:1918] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:37:44.862514 26097 log.cc:1196] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/3f9a835a53c74b65945274c81b05bcff
I20260806 20:37:44.863355 26097 ts_tablet_manager.cc:1939] T 3f9a835a53c74b65945274c81b05bcff P b91c0f1653d2437c82231114723feed5: Deleting consensus metadata
I20260806 20:37:44.865973 25522 catalog_manager.cc:4841] TS b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903): tablet 3f9a835a53c74b65945274c81b05bcff (table test-table [id=1dc4d8f1bc5e4a22a7a5b69c157cf561]) successfully deleted
I20260806 20:37:44.867996 25679 tablet_service.cc:1467] Processing CreateTablet for tablet bdaa7f004db54982a836a07f34d7e461 (DEFAULT_TABLE table=test-table [id=84d16b65220a4714bdd5170fcec4c0f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:44.870869 25679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bdaa7f004db54982a836a07f34d7e461. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:44.872983 26071 tablet_bootstrap.cc:492] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: No bootstrap required, opened a new log
I20260806 20:37:44.873136 26071 ts_tablet_manager.cc:1397] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260806 20:37:44.874008 26071 raft_consensus.cc:348] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:44.874258 26071 raft_consensus.cc:374] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:44.874378 26071 raft_consensus.cc:729] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5588a0d804344ec6aeced87f56165107, State: Initialized, Role: FOLLOWER
I20260806 20:37:44.874735 26071 consensus_queue.cc:260] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:44.875303 26095 ts_tablet_manager.cc:1918] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:37:44.875545 26095 log.cc:1196] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/3f9a835a53c74b65945274c81b05bcff
I20260806 20:37:44.876191 26071 ts_tablet_manager.cc:1428] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:44.877585 26095 ts_tablet_manager.cc:1939] T 3f9a835a53c74b65945274c81b05bcff P 5588a0d804344ec6aeced87f56165107: Deleting consensus metadata
I20260806 20:37:44.881765 25522 catalog_manager.cc:4841] TS 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821): tablet 3f9a835a53c74b65945274c81b05bcff (table test-table [id=1dc4d8f1bc5e4a22a7a5b69c157cf561]) successfully deleted
I20260806 20:37:44.883062 26096 tablet_replica.cc:330] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: stopping tablet replica
I20260806 20:37:44.883632 26069 tablet_bootstrap.cc:492] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Bootstrap starting.
I20260806 20:37:44.883620 26096 raft_consensus.cc:2229] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:37:44.884693 26096 raft_consensus.cc:2258] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:44.885478 26069 tablet_bootstrap.cc:654] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:44.886052 26070 tablet_bootstrap.cc:492] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Bootstrap starting.
I20260806 20:37:44.888446 26070 tablet_bootstrap.cc:654] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:44.888212 26096 ts_tablet_manager.cc:1905] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:44.890581 26069 tablet_bootstrap.cc:492] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: No bootstrap required, opened a new log
I20260806 20:37:44.890810 26069 ts_tablet_manager.cc:1397] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260806 20:37:44.892014 26069 raft_consensus.cc:348] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:44.892279 26069 raft_consensus.cc:374] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:44.892401 26069 raft_consensus.cc:729] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91c0f1653d2437c82231114723feed5, State: Initialized, Role: FOLLOWER
I20260806 20:37:44.892761 26069 consensus_queue.cc:260] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:44.895741 26069 ts_tablet_manager.cc:1428] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 20:37:44.898388 26096 ts_tablet_manager.cc:1918] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:37:44.898563 26096 log.cc:1196] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/3f9a835a53c74b65945274c81b05bcff
I20260806 20:37:44.899511 26096 ts_tablet_manager.cc:1939] T 3f9a835a53c74b65945274c81b05bcff P c7fac236ac204a06b96b662c83708eed: Deleting consensus metadata
I20260806 20:37:44.899881 26070 tablet_bootstrap.cc:492] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: No bootstrap required, opened a new log
I20260806 20:37:44.900045 26070 ts_tablet_manager.cc:1397] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.010s
I20260806 20:37:44.900950 26070 raft_consensus.cc:348] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:44.901155 25519 catalog_manager.cc:4841] TS c7fac236ac204a06b96b662c83708eed (127.23.200.1:39317): tablet 3f9a835a53c74b65945274c81b05bcff (table test-table [id=1dc4d8f1bc5e4a22a7a5b69c157cf561]) successfully deleted
I20260806 20:37:44.901183 26070 raft_consensus.cc:374] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:44.901348 26070 raft_consensus.cc:729] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fac236ac204a06b96b662c83708eed, State: Initialized, Role: FOLLOWER
I20260806 20:37:44.901676 26070 consensus_queue.cc:260] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:44.902462 26070 ts_tablet_manager.cc:1428] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Time spent starting tablet: real 0.002s user 0.002s sys 0.002s
I20260806 20:37:45.064216 26091 raft_consensus.cc:482] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:45.064544 26091 raft_consensus.cc:504] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.065436 26091 leader_election.cc:288] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903), 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821)
I20260806 20:37:45.066059 25854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaa7f004db54982a836a07f34d7e461" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91c0f1653d2437c82231114723feed5" is_pre_election: true
I20260806 20:37:45.066151 25997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaa7f004db54982a836a07f34d7e461" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5588a0d804344ec6aeced87f56165107" is_pre_election: true
I20260806 20:37:45.066310 25854 raft_consensus.cc:2454] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 0.
I20260806 20:37:45.066509 25997 raft_consensus.cc:2454] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 0.
I20260806 20:37:45.066856 25630 leader_election.cc:302] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b91c0f1653d2437c82231114723feed5, c7fac236ac204a06b96b662c83708eed; no voters:
I20260806 20:37:45.067176 26091 raft_consensus.cc:2781] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:45.067324 26091 raft_consensus.cc:482] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:45.067523 26091 raft_consensus.cc:3037] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:45.070281 26091 raft_consensus.cc:504] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.071136 26091 leader_election.cc:288] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 election: Requested vote from peers b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903), 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821)
I20260806 20:37:45.071394 25854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaa7f004db54982a836a07f34d7e461" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91c0f1653d2437c82231114723feed5"
I20260806 20:37:45.071621 25854 raft_consensus.cc:3037] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:45.071614 25997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bdaa7f004db54982a836a07f34d7e461" candidate_uuid: "c7fac236ac204a06b96b662c83708eed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5588a0d804344ec6aeced87f56165107"
I20260806 20:37:45.071846 25997 raft_consensus.cc:3037] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:45.074239 25997 raft_consensus.cc:2454] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 1.
I20260806 20:37:45.074816 25630 leader_election.cc:302] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5588a0d804344ec6aeced87f56165107, c7fac236ac204a06b96b662c83708eed; no voters:
I20260806 20:37:45.075054 25854 raft_consensus.cc:2454] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7fac236ac204a06b96b662c83708eed in term 1.
I20260806 20:37:45.075115 26091 raft_consensus.cc:2781] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:45.075294 26091 raft_consensus.cc:686] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 1 LEADER]: Becoming Leader. State: Replica: c7fac236ac204a06b96b662c83708eed, State: Running, Role: LEADER
I20260806 20:37:45.075713 26091 consensus_queue.cc:237] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.078794 25535 catalog_manager.cc:5495] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed reported cstate change: term changed from 0 to 1, leader changed from <none> to c7fac236ac204a06b96b662c83708eed (127.23.200.1). New cstate: current_term: 1 leader_uuid: "c7fac236ac204a06b96b662c83708eed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } health_report { overall_health: UNKNOWN } } }
Aug 06 20:37:45 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048662, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:45.139775 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.124592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35337 (local address 127.0.0.1:38116)
0806 20:37:45.125292 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:45.125322 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:45.125439 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:45.126047 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:45.126051 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:45.126852 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:45.127068 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.127077 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.128139 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.128145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.129139 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.129151 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.129887 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.129893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.130080 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.130951 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:45.130968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:45.133447 (+ 2479us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:45.137043 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:45.137055 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:45.137063 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:45.137450 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:45.137852 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:45.137856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:45.137860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:45.138000 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:45.138631 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:45.138639 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:45.138796 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 20:37:45.138978 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 20:37:45.139252 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":404,"threads_started":1}
I20260806 20:37:45.139890 26109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.124672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38116 (local address 127.23.200.62:35337)
0806 20:37:45.125801 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.125805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.125821 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:37:45.125856 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.125858 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.125906 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.126021 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:45.127250 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.127998 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.129292 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.129706 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.129847 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.133628 (+ 3781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:45.133633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:45.133636 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:45.133662 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:45.136903 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.137564 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.137571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.137574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.137736 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.138101 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.138107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.138111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.138508 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:45.138656 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.139065 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.139374 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":1044,"thread_start_us":939,"threads_started":1}
I20260806 20:37:45.145226 25535 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38116:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 20:37:45.146436 25535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:45.153901 25679 tablet_service.cc:1467] Processing CreateTablet for tablet 496f05ee463e4b609dceddeb9c36909b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bde33bee76634854a7e8677fec0dca07]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:37:45.154040 25832 tablet_service.cc:1467] Processing CreateTablet for tablet 496f05ee463e4b609dceddeb9c36909b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bde33bee76634854a7e8677fec0dca07]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:37:45.154444 25973 tablet_service.cc:1467] Processing CreateTablet for tablet 496f05ee463e4b609dceddeb9c36909b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bde33bee76634854a7e8677fec0dca07]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:37:45.154686 25679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496f05ee463e4b609dceddeb9c36909b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:45.154731 25832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496f05ee463e4b609dceddeb9c36909b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:45.155031 25973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 496f05ee463e4b609dceddeb9c36909b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:45.162963 26069 tablet_bootstrap.cc:492] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: Bootstrap starting.
I20260806 20:37:45.164175 26071 tablet_bootstrap.cc:492] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: Bootstrap starting.
I20260806 20:37:45.165177 26069 tablet_bootstrap.cc:654] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:45.165532 26070 tablet_bootstrap.cc:492] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: Bootstrap starting.
I20260806 20:37:45.166241 26071 tablet_bootstrap.cc:654] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:45.167518 26070 tablet_bootstrap.cc:654] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:45.176383 26071 tablet_bootstrap.cc:492] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: No bootstrap required, opened a new log
I20260806 20:37:45.176573 26071 ts_tablet_manager.cc:1397] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20260806 20:37:45.177170 26071 raft_consensus.cc:348] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.177395 26071 raft_consensus.cc:374] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:45.177506 26071 raft_consensus.cc:729] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5588a0d804344ec6aeced87f56165107, State: Initialized, Role: FOLLOWER
I20260806 20:37:45.177780 26071 consensus_queue.cc:260] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.178452 26075 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.178776 26075 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.179198 26071 ts_tablet_manager.cc:1428] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 20:37:45.180338 26069 tablet_bootstrap.cc:492] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: No bootstrap required, opened a new log
I20260806 20:37:45.180572 26069 ts_tablet_manager.cc:1397] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.010s
I20260806 20:37:45.181545 26069 raft_consensus.cc:348] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.181901 26069 raft_consensus.cc:374] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:45.182133 26069 raft_consensus.cc:729] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b91c0f1653d2437c82231114723feed5, State: Initialized, Role: FOLLOWER
I20260806 20:37:45.182597 26069 consensus_queue.cc:260] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.183336 26069 ts_tablet_manager.cc:1428] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 20:37:45.183368 26070 tablet_bootstrap.cc:492] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: No bootstrap required, opened a new log
I20260806 20:37:45.183173 26076 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.183563 26070 ts_tablet_manager.cc:1397] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.010s
I20260806 20:37:45.183609 26076 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.185521 26070 raft_consensus.cc:348] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.185789 26070 raft_consensus.cc:374] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:45.185931 26070 raft_consensus.cc:729] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7fac236ac204a06b96b662c83708eed, State: Initialized, Role: FOLLOWER
I20260806 20:37:45.186210 26070 consensus_queue.cc:260] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.186733 26091 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.186932 26070 ts_tablet_manager.cc:1428] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 20:37:45.187042 26091 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.459614 26091 consensus_queue.cc:1035] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:45.464628 26077 consensus_queue.cc:1035] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:45.491063 26075 raft_consensus.cc:482] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:45.491377 26075 raft_consensus.cc:504] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.492985 26075 leader_election.cc:288] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7fac236ac204a06b96b662c83708eed (127.23.200.1:39317), b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903)
I20260806 20:37:45.511884 26119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.493193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39317 (local address 127.23.200.3:39849)
0806 20:37:45.505881 (+ 12688us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:45.505907 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:45.506085 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:45.506599 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:45.506603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:45.506632 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:45.506896 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.506905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.507959 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.507968 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.509419 (+ 1451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.509431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.510973 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.510979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.511214 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.511244 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 20:37:45.511375 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 20:37:45.511509 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12528,"thread_start_us":12412,"threads_started":1}
I20260806 20:37:45.511937 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.493290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:39849 (local address 127.23.200.1:39317)
0806 20:37:45.506224 (+ 12934us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.506232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.506253 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:37:45.506386 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.506390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.506457 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.506563 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:45.507053 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.507810 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.509590 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.510790 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.510973 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.511052 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.511403 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.511550 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12784,"thread_start_us":12658,"threads_started":1}
I20260806 20:37:45.512102 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.493516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:36519 (local address 127.23.200.2:34903)
0806 20:37:45.505703 (+ 12187us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.505712 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.506628 (+ 916us) server_negotiation.cc:407] Connection header received
0806 20:37:45.506846 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.506850 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.506907 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.507012 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:45.507760 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.508477 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.510085 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.511022 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.511224 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.511298 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.511602 (+ 304us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.511741 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11976,"thread_start_us":11818,"threads_started":1}
I20260806 20:37:45.512140 26120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.493248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:34903 (local address 127.23.200.3:36519)
0806 20:37:45.506508 (+ 13260us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:45.506523 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:45.506691 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:45.507027 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:45.507031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:45.507055 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:45.507527 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.507544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.508622 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.508630 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.509916 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.509927 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.511244 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.511251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.511424 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.511448 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 20:37:45.511573 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 20:37:45.511706 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13149,"thread_start_us":12970,"threads_started":1}
I20260806 20:37:45.513123 25854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496f05ee463e4b609dceddeb9c36909b" candidate_uuid: "5588a0d804344ec6aeced87f56165107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91c0f1653d2437c82231114723feed5" is_pre_election: true
I20260806 20:37:45.513303 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496f05ee463e4b609dceddeb9c36909b" candidate_uuid: "5588a0d804344ec6aeced87f56165107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fac236ac204a06b96b662c83708eed" is_pre_election: true
I20260806 20:37:45.513461 25854 raft_consensus.cc:2454] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5588a0d804344ec6aeced87f56165107 in term 0.
I20260806 20:37:45.513797 25707 raft_consensus.cc:2454] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5588a0d804344ec6aeced87f56165107 in term 0.
I20260806 20:37:45.514027 25926 leader_election.cc:302] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5588a0d804344ec6aeced87f56165107, b91c0f1653d2437c82231114723feed5; no voters:
I20260806 20:37:45.514444 26075 raft_consensus.cc:2781] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:45.514581 26075 raft_consensus.cc:482] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:45.514696 26075 raft_consensus.cc:3037] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:45.517017 26075 raft_consensus.cc:504] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.517663 26075 leader_election.cc:288] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [CANDIDATE]: Term 1 election: Requested vote from peers c7fac236ac204a06b96b662c83708eed (127.23.200.1:39317), b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903)
I20260806 20:37:45.518077 25707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496f05ee463e4b609dceddeb9c36909b" candidate_uuid: "5588a0d804344ec6aeced87f56165107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7fac236ac204a06b96b662c83708eed"
I20260806 20:37:45.518087 25854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "496f05ee463e4b609dceddeb9c36909b" candidate_uuid: "5588a0d804344ec6aeced87f56165107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b91c0f1653d2437c82231114723feed5"
I20260806 20:37:45.518286 25707 raft_consensus.cc:3037] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:45.518373 25854 raft_consensus.cc:3037] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:45.521750 25707 raft_consensus.cc:2454] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5588a0d804344ec6aeced87f56165107 in term 1.
I20260806 20:37:45.522023 25854 raft_consensus.cc:2454] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5588a0d804344ec6aeced87f56165107 in term 1.
I20260806 20:37:45.522220 25923 leader_election.cc:302] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5588a0d804344ec6aeced87f56165107, c7fac236ac204a06b96b662c83708eed; no voters:
I20260806 20:37:45.522599 26075 raft_consensus.cc:2781] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:45.523036 26075 raft_consensus.cc:686] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [term 1 LEADER]: Becoming Leader. State: Replica: 5588a0d804344ec6aeced87f56165107, State: Running, Role: LEADER
I20260806 20:37:45.523415 26075 consensus_queue.cc:237] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } }
I20260806 20:37:45.525543 26123 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: TxnStatusTablet state changed. Reason: New leader 5588a0d804344ec6aeced87f56165107. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.526088 26123 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:37:45.527163 26126 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 20:37:45.527385 26126 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 20:37:45.527380 25533 catalog_manager.cc:5495] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5588a0d804344ec6aeced87f56165107 (127.23.200.3). New cstate: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } health_report { overall_health: HEALTHY } } }
I20260806 20:37:45.632110 26108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.624348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:32821 (local address 127.0.0.1:56660)
0806 20:37:45.624619 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:45.624641 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:45.624831 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:45.625855 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:45.625860 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:45.625872 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:45.626117 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.626130 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.627504 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.627511 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.628711 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.628722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.629708 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.629716 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.630033 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.630089 (+ 56us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:45.631211 (+ 1122us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:45.631234 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:37:45.631325 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 20:37:45.631482 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 20:37:45.632187 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.624369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56660 (local address 127.23.200.3:32821)
0806 20:37:45.625479 (+ 1110us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.625484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.625502 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:37:45.625581 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.625584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.625660 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.625834 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:45.626394 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.627357 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.628869 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.629495 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.629648 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.630381 (+ 733us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:45.631090 (+ 709us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:45.631259 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.631376 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.631463 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":742,"threads_started":1}
I20260806 20:37:45.636983 25854 raft_consensus.cc:1261] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Refusing update from remote peer 5588a0d804344ec6aeced87f56165107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:45.637059 25707 raft_consensus.cc:1261] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Refusing update from remote peer 5588a0d804344ec6aeced87f56165107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:45.637804 26075 consensus_queue.cc:1035] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:45.638231 26123 consensus_queue.cc:1035] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:45.642163 26076 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: TxnStatusTablet state changed. Reason: New leader 5588a0d804344ec6aeced87f56165107. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.642568 26076 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.645107 26077 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: TxnStatusTablet state changed. Reason: New leader 5588a0d804344ec6aeced87f56165107. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.645547 26077 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.645689 26118 mvcc.cc:205] Tried to move back new op lower bound from 7315655334440738816 to 7315655333987618816. Current Snapshot: MvccSnapshot[applied={T|T < 7315655334440738816}]
I20260806 20:37:45.648648 26116 mvcc.cc:205] Tried to move back new op lower bound from 7315655334440738816 to 7315655333987618816. Current Snapshot: MvccSnapshot[applied={T|T < 7315655334440738816}]
I20260806 20:37:45.650676 26123 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.651266 26076 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.651605 26076 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P b91c0f1653d2437c82231114723feed5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.653012 26123 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:37:45.654013 26123 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.654366 26123 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P 5588a0d804344ec6aeced87f56165107: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:37:45.661834 26077 tablet_replica.cc:439] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5588a0d804344ec6aeced87f56165107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7fac236ac204a06b96b662c83708eed" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39317 } } peers { permanent_uuid: "b91c0f1653d2437c82231114723feed5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 34903 } } peers { permanent_uuid: "5588a0d804344ec6aeced87f56165107" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 32821 } } }
I20260806 20:37:45.662156 26077 tablet_replica.cc:442] T 496f05ee463e4b609dceddeb9c36909b P c7fac236ac204a06b96b662c83708eed: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:45.692656 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.685117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:45523 (local address 127.23.200.3:32821)
0806 20:37:45.685290 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.685310 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.685965 (+ 655us) server_negotiation.cc:407] Connection header received
0806 20:37:45.686151 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.686157 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.686237 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.686358 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:45.687064 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.688389 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.690403 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.691087 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.691219 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.691861 (+ 642us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:45.692056 (+ 195us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:45.692148 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.692306 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.692387 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:37:45.693132 26139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.685066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:32821 (local address 127.23.200.1:45523)
0806 20:37:45.685832 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:45.685858 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:45.686016 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:45.686399 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:45.686406 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:45.686423 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:45.686903 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.686916 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.688539 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.688545 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.690212 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.690228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.691278 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.691285 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.691568 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.691637 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:45.692173 (+ 536us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:45.692186 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 20:37:45.692276 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 20:37:45.692450 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":448,"threads_started":1}
I20260806 20:37:45.733982 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.726400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:59815 (local address 127.23.200.1:39317)
0806 20:37:45.726591 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.726598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.727263 (+ 665us) server_negotiation.cc:407] Connection header received
0806 20:37:45.727434 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.727448 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.727529 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.727641 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:45.728194 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.729299 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.731746 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.732380 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.732528 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.733236 (+ 708us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:45.733365 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:45.733467 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.733587 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.733685 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:37:45.734014 26142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.726322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39317 (local address 127.23.200.3:59815)
0806 20:37:45.727162 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:45.727179 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:45.727314 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:45.727642 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:45.727646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:45.727655 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:45.728011 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.728019 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.729496 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.729508 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.731550 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:45.731570 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.732662 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.732675 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:45.732947 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.733028 (+ 81us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:45.733467 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:45.733477 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 20:37:45.733571 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 20:37:45.733730 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":345,"threads_started":1}
I20260806 20:37:45.773075 25533 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:45.773460 25533 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:45.776624 25533 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 717d1480f7db4cda978244b0451e2f65: Sending DeleteTablet for 3 replicas of tablet bdaa7f004db54982a836a07f34d7e461
I20260806 20:37:45.777390 25679 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaa7f004db54982a836a07f34d7e461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:45 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:37704
I20260806 20:37:45.777587 25832 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaa7f004db54982a836a07f34d7e461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:45 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35168
I20260806 20:37:45.778264 25973 tablet_service.cc:1514] Processing DeleteTablet for tablet bdaa7f004db54982a836a07f34d7e461 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:45 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56654
I20260806 20:37:45.788357 26144 tablet_replica.cc:330] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: stopping tablet replica
I20260806 20:37:45.788678 26144 raft_consensus.cc:2229] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:37:45.789179 26147 tablet_replica.cc:330] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: stopping tablet replica
I20260806 20:37:45.789250 26146 tablet_replica.cc:330] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: stopping tablet replica
I20260806 20:37:45.789188 26144 raft_consensus.cc:2258] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:45.789408 26147 raft_consensus.cc:2229] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:45.789456 26146 raft_consensus.cc:2229] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:45.789708 26147 raft_consensus.cc:2258] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:45.789741 26146 raft_consensus.cc:2258] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:45.790891 26147 ts_tablet_manager.cc:1905] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:45.790906 26144 ts_tablet_manager.cc:1905] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:45.794430 26146 ts_tablet_manager.cc:1905] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:45.799801 26144 ts_tablet_manager.cc:1918] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:45.800149 26144 log.cc:1196] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/bdaa7f004db54982a836a07f34d7e461
I20260806 20:37:45.800781 26144 ts_tablet_manager.cc:1939] T bdaa7f004db54982a836a07f34d7e461 P c7fac236ac204a06b96b662c83708eed: Deleting consensus metadata
I20260806 20:37:45.802039 25519 catalog_manager.cc:4841] TS c7fac236ac204a06b96b662c83708eed (127.23.200.1:39317): tablet bdaa7f004db54982a836a07f34d7e461 (table test-table [id=84d16b65220a4714bdd5170fcec4c0f2]) successfully deleted
I20260806 20:37:45.804741 26147 ts_tablet_manager.cc:1918] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:45.804922 26147 log.cc:1196] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/bdaa7f004db54982a836a07f34d7e461
I20260806 20:37:45.805459 26147 ts_tablet_manager.cc:1939] T bdaa7f004db54982a836a07f34d7e461 P 5588a0d804344ec6aeced87f56165107: Deleting consensus metadata
I20260806 20:37:45.806694 25522 catalog_manager.cc:4841] TS 5588a0d804344ec6aeced87f56165107 (127.23.200.3:32821): tablet bdaa7f004db54982a836a07f34d7e461 (table test-table [id=84d16b65220a4714bdd5170fcec4c0f2]) successfully deleted
I20260806 20:37:45.809980 26146 ts_tablet_manager.cc:1918] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:45.810199 26146 log.cc:1196] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/bdaa7f004db54982a836a07f34d7e461
I20260806 20:37:45.810801 26146 ts_tablet_manager.cc:1939] T bdaa7f004db54982a836a07f34d7e461 P b91c0f1653d2437c82231114723feed5: Deleting consensus metadata
I20260806 20:37:45.811954 25522 catalog_manager.cc:4841] TS b91c0f1653d2437c82231114723feed5 (127.23.200.2:34903): tablet bdaa7f004db54982a836a07f34d7e461 (table test-table [id=84d16b65220a4714bdd5170fcec4c0f2]) successfully deleted
Aug 06 20:37:45 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:45.822445 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.788192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37714 (local address 127.23.200.1:39317)
0806 20:37:45.788391 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.788398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.793411 (+ 5013us) server_negotiation.cc:407] Connection header received
0806 20:37:45.793799 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.793804 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.793886 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.793971 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:45.796879 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.798171 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.800370 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.801098 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.801193 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.814262 (+ 13069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:45.814280 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:45.814299 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:45.814416 (+ 117us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:45.818820 (+ 4404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.819707 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.819719 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.819724 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.819918 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.820403 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.820409 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.820412 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.821020 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:45.821315 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.821661 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.821911 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260806 20:37:45.823688 25749 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37714
I20260806 20:37:45.855636 26153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.827813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.23.200.62:35337)
0806 20:37:45.839448 (+ 11635us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.839453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.839469 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:37:45.839520 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.839523 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.839581 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.839701 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:45.841395 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.842901 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.844329 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.845390 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.845525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.848065 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:45.848072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:45.848077 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:45.848117 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:45.852419 (+ 4302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.853170 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.853175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.853179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.853336 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.853727 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.853731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.853734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.854079 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:45.854372 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.854712 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.854974 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11505,"thread_start_us":11316,"threads_started":1}
W20260806 20:37:45.856671 25533 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38122
I20260806 20:37:45.858907 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25609
I20260806 20:37:45.875052 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.860053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37716 (local address 127.23.200.1:39317)
0806 20:37:45.860236 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.860241 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.860473 (+ 232us) server_negotiation.cc:407] Connection header received
0806 20:37:45.860583 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.860589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.860679 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.860784 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:45.861933 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.863273 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.864516 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.865403 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.865547 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.868287 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:45.868293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:45.868298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:45.868338 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:45.872244 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.872893 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.872899 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.872903 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.873071 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.873480 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.873486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.873490 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.873856 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:45.874139 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.874405 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.874648 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 20:37:45.875635 25749 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37716
W20260806 20:37:45.876174 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25609: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:37:45.876430 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25609
I20260806 20:37:45.901331 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25759
Aug 06 20:37:45 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:37:45.917729 26122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.902182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35172 (local address 127.23.200.2:34903)
0806 20:37:45.902423 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.902429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.902453 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:37:45.902665 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.902671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.902763 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.902887 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:45.904012 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.905093 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.906539 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.907254 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.907396 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.911760 (+ 4364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:45.911775 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:45.911790 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:45.911830 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:45.914999 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.915804 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.915810 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.915813 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.915970 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.916368 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.916371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.916373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.916721 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:45.916937 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.917280 (+ 343us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.917454 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 20:37:45.918573 25895 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35172
W20260806 20:37:45.919267 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25759: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:37:45.919418 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25759
I20260806 20:37:45.939253 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25905
Aug 06 20:37:45 dist-test-slave-tm1g krb5kdc[25489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048664, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:37:45.956826 26129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:45.940084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.23.200.3:32821)
0806 20:37:45.940375 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:45.940381 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:45.940405 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:37:45.940581 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:45.940592 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:45.940678 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:45.940815 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:45.942036 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.943233 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.944499 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:45.945229 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:45.945378 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:45.949884 (+ 4506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:45.949897 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:45.949917 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:45.949957 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:45.954008 (+ 4051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.954703 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.954713 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.954716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.954876 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:45.955229 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:45.955234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:45.955237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:45.955683 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:45.955991 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:45.956157 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 20:37:45.956389 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260806 20:37:45.957708 26044 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56674
W20260806 20:37:45.958496 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25905: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:37:45.958686 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25905
I20260806 20:37:45.980365 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 25503
I20260806 20:37:45.981519 25577 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:46.099990 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 25503
2026-08-06T20:37:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5138 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:37:46 dist-test-slave-tm1g krb5kdc[26159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:37:46 dist-test-slave-tm1g krb5kdc[26159](info): set up 2 sockets
Aug 06 20:37:46 dist-test-slave-tm1g krb5kdc[26159](info): commencing operation
krb5kdc: starting...
W20260806 20:37:47.197736 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.010s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:37:47 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:37:47.379913 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45099
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:45099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:47.697034 26173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:47.697376 26173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:47.697559 26173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:47.704560 26173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:37:47.704700 26173 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 20:37:47.704797 26173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:47.704914 26173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:47.705011 26173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:37:47.705101 26173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:37:47.714541 26173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:45099
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:47.715780 26173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:47.717123 26173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:47.728670 26178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:47.728715 26179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:47.728756 26181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:47.729696 26173 server_base.cc:1034] running on GCE node
I20260806 20:37:47.730508 26173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:47.731812 26173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:47.733028 26173 hybrid_clock.cc:648] HybridClock initialized: now 1786048667732975 us; error 26 us; skew 500 ppm
I20260806 20:37:47.733354 26173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:47 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:47.736598 26173 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:37:47.738039 26173 webserver.cc:460] Webserver started at http://127.23.200.62:43389/ using document root <none> and password file <none>
I20260806 20:37:47.738807 26173 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:47.738921 26173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:47.739269 26173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:47.743206 26173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "9f62a014d91a439c87517d0d27f789ed"
format_stamp: "Formatted at 2026-08-06 20:37:47 on dist-test-slave-tm1g"
server_key: "abce5df15e428894a1ffc307a32bc04d"
server_key_iv: "0eee508977b94991c8f4068e299455ec"
server_key_version: "encryptionkey@0"
I20260806 20:37:47.744279 26173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "9f62a014d91a439c87517d0d27f789ed"
format_stamp: "Formatted at 2026-08-06 20:37:47 on dist-test-slave-tm1g"
server_key: "abce5df15e428894a1ffc307a32bc04d"
server_key_iv: "0eee508977b94991c8f4068e299455ec"
server_key_version: "encryptionkey@0"
I20260806 20:37:47.750123 26173 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 20:37:47.753418 26188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:47.754761 26173 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:47.754940 26173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "9f62a014d91a439c87517d0d27f789ed"
format_stamp: "Formatted at 2026-08-06 20:37:47 on dist-test-slave-tm1g"
server_key: "abce5df15e428894a1ffc307a32bc04d"
server_key_iv: "0eee508977b94991c8f4068e299455ec"
server_key_version: "encryptionkey@0"
I20260806 20:37:47.755148 26173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:47.773912 26173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:47.778275 26173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:47.778633 26173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:47.800374 26173 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:45099
I20260806 20:37:47.800403 26253 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:45099 every 8 connection(s)
I20260806 20:37:47.801733 26173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:37:47.805469 26254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:47.805799 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 26173
I20260806 20:37:47.806056 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:37:47.806891 24352 external_mini_cluster.cc:1428] Setting key 81e477db7468a2be8bd5e92d8901ea67
I20260806 20:37:47.818655 26254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Bootstrap starting.
Aug 06 20:37:47 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:47.821689 26254 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:47.822839 26254 log.cc:824] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:47.825610 26254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: No bootstrap required, opened a new log
I20260806 20:37:47.829653 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:47.808982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.23.200.62:45099)
0806 20:37:47.809790 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:47.809803 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:47.809843 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:37:47.811859 (+ 2016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:47.811880 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:47.812272 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:47.812601 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:47.814404 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:47.815680 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:47.817014 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:47.817668 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:47.817870 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:47.821890 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:47.821899 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:47.821913 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:47.821956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:47.825099 (+ 3143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:47.825806 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:47.825814 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:47.825822 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:47.826075 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:47.826517 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:47.826521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:47.826524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:47.827216 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:47.827513 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:47.827874 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 20:37:47.828319 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":387,"threads_started":1}
I20260806 20:37:47.834378 26254 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62a014d91a439c87517d0d27f789ed" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45099 } }
I20260806 20:37:47.834707 26254 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:47.834779 26254 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f62a014d91a439c87517d0d27f789ed, State: Initialized, Role: FOLLOWER
I20260806 20:37:47.835237 26254 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62a014d91a439c87517d0d27f789ed" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45099 } }
I20260806 20:37:47.835438 26254 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:37:47.835546 26254 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:37:47.835661 26254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:47.837956 26254 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62a014d91a439c87517d0d27f789ed" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45099 } }
I20260806 20:37:47.838477 26254 leader_election.cc:302] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f62a014d91a439c87517d0d27f789ed; no voters:
I20260806 20:37:47.839071 26254 leader_election.cc:288] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:37:47.839170 26260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:47.840233 26260 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [term 1 LEADER]: Becoming Leader. State: Replica: 9f62a014d91a439c87517d0d27f789ed, State: Running, Role: LEADER
I20260806 20:37:47.840735 26260 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62a014d91a439c87517d0d27f789ed" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45099 } }
I20260806 20:37:47.841114 26254 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:37:47.843982 26261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f62a014d91a439c87517d0d27f789ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62a014d91a439c87517d0d27f789ed" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45099 } } }
I20260806 20:37:47.844375 26261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:47.843921 26262 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f62a014d91a439c87517d0d27f789ed. Latest consensus state: current_term: 1 leader_uuid: "9f62a014d91a439c87517d0d27f789ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f62a014d91a439c87517d0d27f789ed" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45099 } } }
I20260806 20:37:47.844982 26262 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:47.851820 26266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:37:47.856653 26266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:37:47.865283 26266 catalog_manager.cc:1347] Generated new cluster ID: 5ed0c1de3bf8440fb16f07c5ee2afd41
I20260806 20:37:47.865412 26266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:37:47.872841 26266 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:37:47.874619 26266 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:37:47.883123 26266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Generated new TSK 0
I20260806 20:37:47.883890 26266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:37:47.938236 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45099
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:48.261639 26281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:48.261935 26281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:48.262104 26281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:48.269217 26281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:48.269372 26281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:48.269521 26281 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 20:37:48.269608 26281 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:37:48.269731 26281 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:37:48.279452 26281 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:48.280751 26281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:48.282109 26281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:48.295056 26287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:48.295095 26286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:48.295295 26289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:48.296151 26281 server_base.cc:1034] running on GCE node
I20260806 20:37:48.296751 26281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:48.297649 26281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:48.298883 26281 hybrid_clock.cc:648] HybridClock initialized: now 1786048668298853 us; error 42 us; skew 500 ppm
I20260806 20:37:48.299201 26281 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:48 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048668, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:48.302255 26281 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:37:48.303825 26281 webserver.cc:460] Webserver started at http://127.23.200.1:33465/ using document root <none> and password file <none>
I20260806 20:37:48.304497 26281 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:48.304648 26281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:48.304962 26281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:48.307989 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565"
format_stamp: "Formatted at 2026-08-06 20:37:48 on dist-test-slave-tm1g"
server_key: "63e4e790456de5c439b79adc40b0dde3"
server_key_iv: "d397b01b6cb154b8b1970fb977131b22"
server_key_version: "encryptionkey@0"
I20260806 20:37:48.308761 26281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565"
format_stamp: "Formatted at 2026-08-06 20:37:48 on dist-test-slave-tm1g"
server_key: "63e4e790456de5c439b79adc40b0dde3"
server_key_iv: "d397b01b6cb154b8b1970fb977131b22"
server_key_version: "encryptionkey@0"
I20260806 20:37:48.313755 26281 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:37:48.317123 26297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:48.318380 26281 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:37:48.318555 26281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565"
format_stamp: "Formatted at 2026-08-06 20:37:48 on dist-test-slave-tm1g"
server_key: "63e4e790456de5c439b79adc40b0dde3"
server_key_iv: "d397b01b6cb154b8b1970fb977131b22"
server_key_version: "encryptionkey@0"
I20260806 20:37:48.318747 26281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:48.339255 26281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:48.343439 26281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:48.343724 26281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:48.346572 26281 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:48.346705 26281 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:48.346863 26281 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:48.346961 26281 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:37:48 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048668, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:48.372722 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.346952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:54543 (local address 127.23.200.62:45099)
0806 20:37:48.347478 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:48.347487 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:48.348460 (+ 973us) server_negotiation.cc:407] Connection header received
0806 20:37:48.350489 (+ 2029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:48.350497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:48.350587 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:48.350710 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:48.353138 (+ 2428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.354352 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.357497 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.358160 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.358343 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.364006 (+ 5663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:48.364017 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:48.364025 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:48.364080 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:48.368010 (+ 3930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.369279 (+ 1269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.369285 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.369288 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.369469 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.370210 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.370218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.370222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.370680 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:48.370861 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:48.371959 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0806 20:37:48.372232 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
I20260806 20:37:48.374610 26308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.347513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.23.200.1:54543)
0806 20:37:48.348295 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:48.348343 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:48.350159 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:48.351010 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:48.351023 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:48.351809 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:48.352874 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.352907 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.354555 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.354567 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.357270 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.357289 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.358498 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.358506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.358928 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.359900 (+ 972us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:48.359943 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:48.363713 (+ 3770us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:48.368178 (+ 4465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.368190 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.368211 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.369140 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.369613 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.369625 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.369631 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.370065 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.370852 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:48.370867 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:48.371450 (+ 583us) client_negotiation.cc:769] Sending connection context
0806 20:37:48.371853 (+ 403us) client_negotiation.cc:240] Negotiation successful
0806 20:37:48.372535 (+ 682us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":329,"threads_started":1}
I20260806 20:37:48.387203 26281 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:33403
I20260806 20:37:48.387300 26425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:33403 every 8 connection(s)
I20260806 20:37:48.388670 26281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:37:48.390967 26426 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45099
I20260806 20:37:48.391197 26426 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:48.391878 26426 heartbeater.cc:511] Master 127.23.200.62:45099 requested a full tablet report, sending...
I20260806 20:37:48.394068 26205 ts_manager.cc:194] Registered new tserver with Master: 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403)
I20260806 20:37:48.396147 26205 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:54543
I20260806 20:37:48.396809 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 26281
I20260806 20:37:48.397038 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:37:48.397636 26415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.382210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.23.200.1:38757)
0806 20:37:48.383000 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:48.383022 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:48.383193 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:48.383583 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:48.383594 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:48.384322 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:48.384801 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.384816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.386376 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.386387 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.388223 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.388241 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.389011 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.389018 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.389231 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.390353 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:48.390376 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:48.391412 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:48.394890 (+ 3478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.394895 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.394899 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.395409 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.395803 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.395808 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.395812 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.395965 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.396570 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:48.396575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:48.396758 (+ 183us) client_negotiation.cc:769] Sending connection context
0806 20:37:48.396909 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 20:37:48.397225 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":413,"threads_started":1}
I20260806 20:37:48.397830 24352 external_mini_cluster.cc:1428] Setting key 49cecdba6f47cfee139db0f66a9af7c9
I20260806 20:37:48.398373 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.382289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38757 (local address 127.23.200.62:45099)
0806 20:37:48.382542 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:48.382551 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:48.383131 (+ 580us) server_negotiation.cc:407] Connection header received
0806 20:37:48.383346 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:48.383354 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:48.383434 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:48.383544 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:48.384976 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.386145 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.388372 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.388826 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.388924 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.391568 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:48.391572 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:48.391576 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:48.391604 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:48.394736 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.395539 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.395543 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.395545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.395699 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.396075 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.396079 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.396081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.396435 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:48.396611 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:48.397954 (+ 1343us) server_negotiation.cc:299] Negotiation successful
0806 20:37:48.398119 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:37:48.438149 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45099
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:48.754730 26430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:48.755041 26430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:48.755218 26430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:48.762557 26430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:48.762704 26430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:48.762850 26430 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 20:37:48.762938 26430 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:37:48.763070 26430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:37:48.772595 26430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:48.773880 26430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:48.775226 26430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:48.788324 26436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:48.788416 26438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:48.788393 26435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:48.791668 26430 server_base.cc:1034] running on GCE node
I20260806 20:37:48.792191 26430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:48.793104 26430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:48.794341 26430 hybrid_clock.cc:648] HybridClock initialized: now 1786048668794268 us; error 29 us; skew 500 ppm
I20260806 20:37:48.794665 26430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:48 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048668, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:48.797720 26430 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:37:48.799201 26430 webserver.cc:460] Webserver started at http://127.23.200.2:44561/ using document root <none> and password file <none>
I20260806 20:37:48.799954 26430 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:48.800101 26430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:48.800406 26430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:48.803301 26430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "c2a4a348653a4492a6bf999b51fd0806"
format_stamp: "Formatted at 2026-08-06 20:37:48 on dist-test-slave-tm1g"
server_key: "bccfbdf9fad3596b0bab12ae462c05e4"
server_key_iv: "3df98ad2e672ed517318266b3a5b3554"
server_key_version: "encryptionkey@0"
I20260806 20:37:48.804041 26430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "c2a4a348653a4492a6bf999b51fd0806"
format_stamp: "Formatted at 2026-08-06 20:37:48 on dist-test-slave-tm1g"
server_key: "bccfbdf9fad3596b0bab12ae462c05e4"
server_key_iv: "3df98ad2e672ed517318266b3a5b3554"
server_key_version: "encryptionkey@0"
I20260806 20:37:48.809103 26430 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:37:48.812628 26445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:48.813936 26430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:37:48.814112 26430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "c2a4a348653a4492a6bf999b51fd0806"
format_stamp: "Formatted at 2026-08-06 20:37:48 on dist-test-slave-tm1g"
server_key: "bccfbdf9fad3596b0bab12ae462c05e4"
server_key_iv: "3df98ad2e672ed517318266b3a5b3554"
server_key_version: "encryptionkey@0"
I20260806 20:37:48.814309 26430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:48.834295 26430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:48.838768 26430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:48.839052 26430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:48.841816 26430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:48.841979 26430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:48.842114 26430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:48.842214 26430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:37:48 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048668, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:48.864449 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.842384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:41103 (local address 127.23.200.62:45099)
0806 20:37:48.842649 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:48.842658 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:48.844226 (+ 1568us) server_negotiation.cc:407] Connection header received
0806 20:37:48.846379 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:48.846389 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:48.846498 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:48.846617 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:48.848745 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.849848 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.852250 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.852701 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.852867 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.857509 (+ 4642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:48.857518 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:48.857523 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:48.857580 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:48.860778 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.861846 (+ 1068us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.861850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.861853 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.861985 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.862576 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.862579 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.862581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.862861 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:48.862982 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:48.864003 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0806 20:37:48.864192 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:37:48.866374 26455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.842938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.23.200.2:41103)
0806 20:37:48.844007 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:48.844067 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:48.846047 (+ 1980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:48.846873 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:48.846883 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:48.847596 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:48.848543 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.848571 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.850014 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.850021 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.852109 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.852121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.852973 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.852982 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.853284 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.853935 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:48.853959 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:48.857203 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:48.860915 (+ 3712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.860927 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.860947 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.861707 (+ 760us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.862098 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.862105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.862112 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.862444 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.862998 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:48.863009 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:48.863529 (+ 520us) client_negotiation.cc:769] Sending connection context
0806 20:37:48.863881 (+ 352us) client_negotiation.cc:240] Negotiation successful
0806 20:37:48.864516 (+ 635us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":423,"threads_started":1}
I20260806 20:37:48.891193 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.872239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:35561 (local address 127.23.200.62:45099)
0806 20:37:48.872424 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:48.872432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:48.874908 (+ 2476us) server_negotiation.cc:407] Connection header received
0806 20:37:48.875132 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:48.875139 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:48.875231 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:48.875338 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:48.876860 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.877879 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.880513 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.881079 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.881235 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.883819 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:48.883829 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:48.883837 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:48.883887 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:48.887829 (+ 3942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.888784 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.888793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.888799 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.888977 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:48.889474 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:48.889481 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:48.889489 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:48.889972 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:48.890147 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:48.890608 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 20:37:48.890832 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:37:48.891330 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:48.872350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.23.200.2:35561)
0806 20:37:48.874807 (+ 2457us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:48.874824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:48.874967 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:48.875400 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:48.875412 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:48.876127 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:48.876612 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.876648 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.878050 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.878062 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.880341 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:48.880362 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:48.881319 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:48.881328 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:48.881609 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:48.882410 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:48.882441 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:48.883566 (+ 1125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:48.887982 (+ 4416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.887989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.887995 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.888615 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.889138 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:48.889144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:48.889150 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:48.889340 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:48.890106 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:48.890111 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:48.890290 (+ 179us) client_negotiation.cc:769] Sending connection context
0806 20:37:48.890502 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 20:37:48.890849 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2311,"thread_start_us":2164,"threads_started":1}
I20260806 20:37:48.904676 26430 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:35151
I20260806 20:37:48.904752 26571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:35151 every 8 connection(s)
I20260806 20:37:48.905916 26430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:37:48.907912 26572 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45099
I20260806 20:37:48.908111 26572 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:48.908667 26572 heartbeater.cc:511] Master 127.23.200.62:45099 requested a full tablet report, sending...
I20260806 20:37:48.909896 26204 ts_manager.cc:194] Registered new tserver with Master: c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151)
I20260806 20:37:48.910827 26204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:41103
I20260806 20:37:48.914445 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 26430
I20260806 20:37:48.914664 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:37:48.915103 24352 external_mini_cluster.cc:1428] Setting key 96e597d3d0f97341218138846c062fce
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:37:48.955600 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45099
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:49.274993 26576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:49.275245 26576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:49.275344 26576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:49.282477 26576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:49.282574 26576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:49.282663 26576 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 20:37:49.282727 26576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:37:49.282783 26576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:37:49.292119 26576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:49.293318 26576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:49.294691 26576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:49.307891 26581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:49.307935 26582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:49.307852 26584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:49.313326 26576 server_base.cc:1034] running on GCE node
I20260806 20:37:49.313874 26576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:49.314805 26576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:49.316013 26576 hybrid_clock.cc:648] HybridClock initialized: now 1786048669315966 us; error 24 us; skew 500 ppm
I20260806 20:37:49.316359 26576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:49 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048669, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:49.319636 26576 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:37:49.321194 26576 webserver.cc:460] Webserver started at http://127.23.200.3:43027/ using document root <none> and password file <none>
I20260806 20:37:49.321900 26576 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:49.322052 26576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:49.322518 26576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:49.325515 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "38eae1ef5d3742bbb6f431ee03191707"
format_stamp: "Formatted at 2026-08-06 20:37:49 on dist-test-slave-tm1g"
server_key: "f6747c67be8abbbe11a770514ab81612"
server_key_iv: "d25286683973855e2cdf29b0050b9c64"
server_key_version: "encryptionkey@0"
I20260806 20:37:49.326267 26576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "38eae1ef5d3742bbb6f431ee03191707"
format_stamp: "Formatted at 2026-08-06 20:37:49 on dist-test-slave-tm1g"
server_key: "f6747c67be8abbbe11a770514ab81612"
server_key_iv: "d25286683973855e2cdf29b0050b9c64"
server_key_version: "encryptionkey@0"
I20260806 20:37:49.331285 26576 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:37:49.334573 26591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:49.335798 26576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:37:49.335978 26576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "38eae1ef5d3742bbb6f431ee03191707"
format_stamp: "Formatted at 2026-08-06 20:37:49 on dist-test-slave-tm1g"
server_key: "f6747c67be8abbbe11a770514ab81612"
server_key_iv: "d25286683973855e2cdf29b0050b9c64"
server_key_version: "encryptionkey@0"
I20260806 20:37:49.336185 26576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:49.372376 26576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:49.376188 26576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:49.376494 26576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:49.379268 26576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:49.379411 26576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:49.379550 26576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:49.379657 26576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:37:49 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048669, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:49.399660 26426 heartbeater.cc:503] Master 127.23.200.62:45099 was elected leader, sending a full tablet report...
I20260806 20:37:49.401722 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.379765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:55129 (local address 127.23.200.62:45099)
0806 20:37:49.380006 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.380015 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.381449 (+ 1434us) server_negotiation.cc:407] Connection header received
0806 20:37:49.383816 (+ 2367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.383825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.383950 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.384078 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:49.386291 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.387092 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.388613 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.389033 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.389172 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.393389 (+ 4217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:49.393398 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:49.393407 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:49.393465 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:49.396843 (+ 3378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.398007 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.398013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.398020 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.398192 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.398854 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.398862 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.398867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.399279 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:49.399480 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.400883 (+ 1403us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.401208 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 20:37:49.402774 26600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.380403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.23.200.3:55129)
0806 20:37:49.381277 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.381328 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.383440 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.384435 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.384443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.385204 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:49.386087 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.386126 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.387254 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.387259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.388455 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.388467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.389262 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.389269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.389484 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.390132 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:49.390168 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:49.393179 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:49.397021 (+ 3842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:49.397037 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:49.397058 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:49.397853 (+ 795us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:49.398359 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:49.398367 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:49.398371 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:49.398705 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:49.399444 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:49.399456 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:49.400306 (+ 850us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.400645 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.401121 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":408,"threads_started":1}
I20260806 20:37:49.431282 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.410543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:43601 (local address 127.23.200.62:45099)
0806 20:37:49.410753 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.410761 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.411491 (+ 730us) server_negotiation.cc:407] Connection header received
0806 20:37:49.411620 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.411628 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.411729 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.411832 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:49.413508 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.415517 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.418061 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.418778 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.418897 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.421691 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:49.421697 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:49.421702 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:49.421743 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:49.426704 (+ 4961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.427944 (+ 1240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.427951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.427956 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.428128 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.428653 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.428660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.428666 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.429071 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:49.429230 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.430457 (+ 1227us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.430762 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 20:37:49.432313 26667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.410585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.23.200.3:43601)
0806 20:37:49.411381 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.411398 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.411507 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.412054 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.412059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.412845 (+ 786us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:49.413337 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.413356 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.415707 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.415713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.417838 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.417862 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.419261 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.419269 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.419483 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.420148 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:49.420180 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:49.421440 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:49.427027 (+ 5587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:49.427034 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:49.427041 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:49.427752 (+ 711us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:49.428288 (+ 536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:49.428295 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:49.428301 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:49.428484 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:49.429313 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:49.429323 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:49.429568 (+ 245us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.429753 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.430191 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":501,"threads_started":1}
I20260806 20:37:49.436672 26576 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37583
I20260806 20:37:49.436820 26714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37583 every 8 connection(s)
I20260806 20:37:49.438290 26576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:37:49.441123 26715 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45099
I20260806 20:37:49.441388 26715 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:49.442126 26715 heartbeater.cc:511] Master 127.23.200.62:45099 requested a full tablet report, sending...
I20260806 20:37:49.443346 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 26576
I20260806 20:37:49.443553 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:37:49.443825 26204 ts_manager.cc:194] Registered new tserver with Master: 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:49.444147 24352 external_mini_cluster.cc:1428] Setting key dc5e564d94a091943b8d5a7b60923c38
I20260806 20:37:49.445148 26204 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:55129
I20260806 20:37:49.447465 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 20:37:49 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 20:37:49 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:49.500891 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.487831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42668 (local address 127.23.200.62:45099)
0806 20:37:49.488072 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.488080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.488273 (+ 193us) server_negotiation.cc:407] Connection header received
0806 20:37:49.488423 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.488427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.488500 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.488587 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:49.489959 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.490880 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.492036 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.492435 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.492547 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.496231 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:49.496236 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:49.496239 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:49.496269 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:49.498675 (+ 2406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.499234 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.499237 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.499239 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.499353 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.499672 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.499675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.499676 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.499928 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:49.500105 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.500348 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.500511 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 20:37:49.504904 26204 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:37:49.511718 26204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:49.546109 26731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.524200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37583 (local address 127.0.0.1:40474)
0806 20:37:49.535823 (+ 11623us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.535847 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.536005 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.537684 (+ 1679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.537691 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.537717 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:49.538111 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.538126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.540501 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.540506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.543168 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.543183 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.545110 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.545116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.545323 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.545371 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.545520 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.545723 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11454,"thread_start_us":15003,"threads_started":1}
I20260806 20:37:49.546134 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.524710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40474 (local address 127.23.200.3:37583)
0806 20:37:49.532304 (+ 7594us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.532316 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.536167 (+ 3851us) server_negotiation.cc:407] Connection header received
0806 20:37:49.536303 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.536315 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.536666 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.536891 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:49.538444 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.540331 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.543353 (+ 3022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.544901 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.545042 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.545271 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.545565 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.545729 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7396,"thread_start_us":7208,"threads_started":1}
I20260806 20:37:49.548271 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.525093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38608 (local address 127.23.200.2:35151)
0806 20:37:49.540900 (+ 15807us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.540908 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.540934 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:37:49.541248 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.541255 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.541526 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.541713 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:49.542324 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.543321 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.546049 (+ 2728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.547226 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.547373 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.547578 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.547826 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.547969 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15652,"thread_start_us":15873,"threads_started":1}
I20260806 20:37:49.548341 26730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.524575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35151 (local address 127.0.0.1:38608)
0806 20:37:49.539481 (+ 14906us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.539497 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.539617 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.541707 (+ 2090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.541711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.541736 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:49.542136 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.542145 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.543504 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.543510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.545836 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.545852 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.547440 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.547447 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.547632 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.547667 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.547773 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.547892 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14803,"mutex_wait_us":5,"thread_start_us":12645,"threads_started":1}
I20260806 20:37:49.548439 26728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.523417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33403 (local address 127.0.0.1:35074)
0806 20:37:49.524293 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.524354 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.524568 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.540836 (+ 16268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.540841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.540869 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:49.541324 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.541338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.543474 (+ 2136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.543480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.545677 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.545690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.547571 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.547580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.547771 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.547808 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.547878 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.548012 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":47,"thread_start_us":339,"threads_started":1}
I20260806 20:37:49.549333 26729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.523640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35074 (local address 127.23.200.1:33403)
0806 20:37:49.540186 (+ 16546us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.540194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.540219 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:37:49.540343 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.540350 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.540575 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.541533 (+ 958us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:49.541732 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.543280 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.545859 (+ 2579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.547307 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.548654 (+ 1347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.548845 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.548920 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.549058 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16397,"thread_start_us":16297,"threads_started":1}
I20260806 20:37:49.550417 26644 tablet_service.cc:1467] Processing CreateTablet for tablet 072b07f1fafd4ed38d7dae248fc9f75f (DEFAULT_TABLE table=test-table [id=e1a6b492b3294b2ea624ad60cdf184d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:49.552085 26644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072b07f1fafd4ed38d7dae248fc9f75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:49.553684 26351 tablet_service.cc:1467] Processing CreateTablet for tablet 072b07f1fafd4ed38d7dae248fc9f75f (DEFAULT_TABLE table=test-table [id=e1a6b492b3294b2ea624ad60cdf184d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:49.553679 26505 tablet_service.cc:1467] Processing CreateTablet for tablet 072b07f1fafd4ed38d7dae248fc9f75f (DEFAULT_TABLE table=test-table [id=e1a6b492b3294b2ea624ad60cdf184d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:49.554991 26505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072b07f1fafd4ed38d7dae248fc9f75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:49.554997 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 072b07f1fafd4ed38d7dae248fc9f75f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:49.563723 26734 tablet_bootstrap.cc:492] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Bootstrap starting.
I20260806 20:37:49.566762 26734 tablet_bootstrap.cc:654] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:49.567849 26734 log.cc:824] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:49.570341 26737 tablet_bootstrap.cc:492] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Bootstrap starting.
I20260806 20:37:49.571244 26734 tablet_bootstrap.cc:492] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: No bootstrap required, opened a new log
I20260806 20:37:49.571588 26734 ts_tablet_manager.cc:1397] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 20:37:49.573712 26737 tablet_bootstrap.cc:654] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:49.574520 26736 tablet_bootstrap.cc:492] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Bootstrap starting.
I20260806 20:37:49.574932 26737 log.cc:824] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:49.577947 26736 tablet_bootstrap.cc:654] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:49.578572 26737 tablet_bootstrap.cc:492] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: No bootstrap required, opened a new log
I20260806 20:37:49.578835 26737 ts_tablet_manager.cc:1397] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260806 20:37:49.579129 26736 log.cc:824] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Log is configured to *not* fsync() on all Append() calls
I20260806 20:37:49.581403 26736 tablet_bootstrap.cc:492] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: No bootstrap required, opened a new log
I20260806 20:37:49.581657 26736 ts_tablet_manager.cc:1397] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 20:37:49.581745 26734 raft_consensus.cc:348] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.582194 26734 raft_consensus.cc:374] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:49.582294 26734 raft_consensus.cc:729] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38eae1ef5d3742bbb6f431ee03191707, State: Initialized, Role: FOLLOWER
I20260806 20:37:49.582926 26734 consensus_queue.cc:260] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.586988 26715 heartbeater.cc:503] Master 127.23.200.62:45099 was elected leader, sending a full tablet report...
I20260806 20:37:49.587627 26734 ts_tablet_manager.cc:1428] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 20:37:49.589942 26737 raft_consensus.cc:348] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.590384 26737 raft_consensus.cc:374] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:49.590493 26737 raft_consensus.cc:729] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a4a348653a4492a6bf999b51fd0806, State: Initialized, Role: FOLLOWER
I20260806 20:37:49.591059 26737 consensus_queue.cc:260] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.592101 26572 heartbeater.cc:503] Master 127.23.200.62:45099 was elected leader, sending a full tablet report...
I20260806 20:37:49.592005 26736 raft_consensus.cc:348] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.592396 26736 raft_consensus.cc:374] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:49.592494 26736 raft_consensus.cc:729] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Initialized, Role: FOLLOWER
I20260806 20:37:49.592763 26737 ts_tablet_manager.cc:1428] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260806 20:37:49.593055 26736 consensus_queue.cc:260] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.594687 26736 ts_tablet_manager.cc:1428] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20260806 20:37:49.641161 26427 tablet.cc:2369] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:37:49.658061 26573 tablet.cc:2369] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:37:49.690507 26716 tablet.cc:2369] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:37:49.832100 26742 raft_consensus.cc:482] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:49.832549 26742 raft_consensus.cc:504] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.834262 26742 leader_election.cc:288] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:49.843779 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.834769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:53121 (local address 127.23.200.3:37583)
0806 20:37:49.835110 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.835117 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.835840 (+ 723us) server_negotiation.cc:407] Connection header received
0806 20:37:49.836113 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.836118 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.836203 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.836305 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:49.837146 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.838235 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.841326 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.842682 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.842849 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.842923 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.843421 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.843541 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260806 20:37:49.843775 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.835185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:33559 (local address 127.23.200.2:35151)
0806 20:37:49.835399 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.835406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.836032 (+ 626us) server_negotiation.cc:407] Connection header received
0806 20:37:49.836169 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.836174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.836249 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.836336 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:49.837123 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.838261 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.841373 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.842682 (+ 1309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.842849 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.842925 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.843387 (+ 462us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.843513 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 20:37:49.844048 26743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.834733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37583 (local address 127.23.200.1:53121)
0806 20:37:49.835699 (+ 966us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.835722 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.835910 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.836352 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.836357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.836442 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:49.837010 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.837024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.838401 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.838408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.841161 (+ 2753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.841177 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.842967 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.842974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.843154 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.843248 (+ 94us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.843345 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.843558 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":747,"mutex_wait_us":238,"spinlock_wait_cycles":21888,"thread_start_us":547,"threads_started":1}
I20260806 20:37:49.844362 26744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.834924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35151 (local address 127.23.200.1:33559)
0806 20:37:49.835908 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.835931 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.836045 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.836352 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.836357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.836384 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:49.836913 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.836932 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.838424 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.838439 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.841162 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.841177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.842932 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.842945 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.843170 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.843253 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.843345 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.843558 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":314,"thread_start_us":639,"threads_started":1}
I20260806 20:37:49.844954 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b07f1fafd4ed38d7dae248fc9f75f" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707" is_pre_election: true
I20260806 20:37:49.845283 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b07f1fafd4ed38d7dae248fc9f75f" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806" is_pre_election: true
I20260806 20:37:49.845423 26669 raft_consensus.cc:2454] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 0.
I20260806 20:37:49.845741 26525 raft_consensus.cc:2454] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 0.
I20260806 20:37:49.846064 26301 leader_election.cc:302] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, 38eae1ef5d3742bbb6f431ee03191707; no voters:
I20260806 20:37:49.846539 26742 raft_consensus.cc:2781] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:49.846690 26742 raft_consensus.cc:482] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:49.846807 26742 raft_consensus.cc:3037] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:49.848948 26742 raft_consensus.cc:504] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.849509 26742 leader_election.cc:288] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 election: Requested vote from peers c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:49.849954 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b07f1fafd4ed38d7dae248fc9f75f" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707"
I20260806 20:37:49.849993 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "072b07f1fafd4ed38d7dae248fc9f75f" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806"
I20260806 20:37:49.850160 26669 raft_consensus.cc:3037] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:49.850173 26525 raft_consensus.cc:3037] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:49.852419 26669 raft_consensus.cc:2454] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 1.
I20260806 20:37:49.852450 26525 raft_consensus.cc:2454] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 1.
I20260806 20:37:49.852806 26301 leader_election.cc:302] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, 38eae1ef5d3742bbb6f431ee03191707; no voters:
I20260806 20:37:49.853142 26742 raft_consensus.cc:2781] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:49.853590 26742 raft_consensus.cc:686] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 LEADER]: Becoming Leader. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Running, Role: LEADER
I20260806 20:37:49.854040 26742 consensus_queue.cc:237] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.858342 26204 catalog_manager.cc:5495] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } health_report { overall_health: UNKNOWN } } }
Aug 06 20:37:49 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048667, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:49.942046 26305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.928597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42678 (local address 127.23.200.62:45099)
0806 20:37:49.928754 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:49.928761 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:49.929279 (+ 518us) server_negotiation.cc:407] Connection header received
0806 20:37:49.929522 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:49.929526 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:49.929593 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:49.929708 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:49.930971 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.931727 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.932936 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.933401 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.933546 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.936645 (+ 3099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:49.936650 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:49.936653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:49.936681 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:49.939203 (+ 2522us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.939895 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.939902 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.939905 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.940036 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:49.940389 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:49.940396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:49.940399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:49.940782 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:49.940995 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:49.941398 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 20:37:49.941737 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":119,"server-negotiator.queue_time_us":50}
I20260806 20:37:49.942090 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:49.928650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45099 (local address 127.0.0.1:42678)
0806 20:37:49.929178 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:49.929196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:49.929380 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:49.929725 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:49.929729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:49.930580 (+ 851us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:49.930834 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.930842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.931849 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.931856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.932790 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:49.932801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:49.933583 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:49.933590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:49.933752 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:49.934400 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:49.934418 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:49.936455 (+ 2037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:49.939339 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:49.939351 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:49.939359 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:49.939740 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:49.940144 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:49.940150 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:49.940154 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:49.940289 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:49.940909 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:49.940917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:49.941120 (+ 203us) client_negotiation.cc:769] Sending connection context
0806 20:37:49.941320 (+ 200us) client_negotiation.cc:240] Negotiation successful
0806 20:37:49.941609 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":280,"threads_started":1}
I20260806 20:37:49.947070 26204 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:42678:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 20:37:49.948091 26204 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:49.954701 26351 tablet_service.cc:1467] Processing CreateTablet for tablet 58763c47817843468a21804e78bb20be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a084f8b0f19404ba52f56f867e64740]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:37:49.954809 26644 tablet_service.cc:1467] Processing CreateTablet for tablet 58763c47817843468a21804e78bb20be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a084f8b0f19404ba52f56f867e64740]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:37:49.954735 26505 tablet_service.cc:1467] Processing CreateTablet for tablet 58763c47817843468a21804e78bb20be (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8a084f8b0f19404ba52f56f867e64740]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:37:49.955650 26644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58763c47817843468a21804e78bb20be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:49.955650 26505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58763c47817843468a21804e78bb20be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:49.955793 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58763c47817843468a21804e78bb20be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:49.961978 26736 tablet_bootstrap.cc:492] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: Bootstrap starting.
I20260806 20:37:49.962845 26734 tablet_bootstrap.cc:492] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: Bootstrap starting.
I20260806 20:37:49.963486 26737 tablet_bootstrap.cc:492] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: Bootstrap starting.
I20260806 20:37:49.963754 26736 tablet_bootstrap.cc:654] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:49.964574 26734 tablet_bootstrap.cc:654] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:49.964810 26737 tablet_bootstrap.cc:654] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:49.966944 26734 tablet_bootstrap.cc:492] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: No bootstrap required, opened a new log
I20260806 20:37:49.967093 26734 ts_tablet_manager.cc:1397] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260806 20:37:49.967182 26737 tablet_bootstrap.cc:492] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: No bootstrap required, opened a new log
I20260806 20:37:49.967326 26737 ts_tablet_manager.cc:1397] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260806 20:37:49.967375 26736 tablet_bootstrap.cc:492] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: No bootstrap required, opened a new log
I20260806 20:37:49.967535 26736 ts_tablet_manager.cc:1397] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:37:49.967904 26734 raft_consensus.cc:348] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.968158 26734 raft_consensus.cc:374] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:49.968045 26737 raft_consensus.cc:348] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.968264 26734 raft_consensus.cc:729] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38eae1ef5d3742bbb6f431ee03191707, State: Initialized, Role: FOLLOWER
I20260806 20:37:49.968334 26737 raft_consensus.cc:374] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:49.968449 26737 raft_consensus.cc:729] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a4a348653a4492a6bf999b51fd0806, State: Initialized, Role: FOLLOWER
I20260806 20:37:49.968370 26736 raft_consensus.cc:348] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.968624 26736 raft_consensus.cc:374] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:49.968513 26734 consensus_queue.cc:260] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.968713 26736 raft_consensus.cc:729] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Initialized, Role: FOLLOWER
I20260806 20:37:49.968690 26737 consensus_queue.cc:260] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.968923 26736 consensus_queue.cc:260] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:49.969147 26734 ts_tablet_manager.cc:1428] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 20:37:49.969033 26740 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:49.969250 26737 ts_tablet_manager.cc:1428] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 20:37:49.969353 26740 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:49.969331 26742 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:49.969578 26742 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:49.969600 26741 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:49.969988 26741 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:49.970031 26736 ts_tablet_manager.cc:1428] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 20:37:50.233989 26741 raft_consensus.cc:482] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:50.234274 26741 raft_consensus.cc:504] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.235909 26741 leader_election.cc:288] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:50.243436 26732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.236371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:51435 (local address 127.23.200.3:37583)
0806 20:37:50.236594 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:50.236604 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:50.237466 (+ 862us) server_negotiation.cc:407] Connection header received
0806 20:37:50.237648 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:50.237654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:50.237740 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:50.237825 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:50.238554 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.239350 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.241426 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.242394 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.242535 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.242636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:50.242997 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 20:37:50.243142 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:37:50.243479 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.236256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37583 (local address 127.23.200.2:51435)
0806 20:37:50.237320 (+ 1064us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:50.237345 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:50.237501 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:50.237866 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:50.237871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:50.237904 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:50.238347 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.238365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.239484 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.239489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.241298 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.241310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.242613 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.242620 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.242835 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.242858 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:37:50.242937 (+ 79us) client_negotiation.cc:240] Negotiation successful
0806 20:37:50.243028 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"mutex_wait_us":96,"thread_start_us":941,"threads_started":1}
I20260806 20:37:50.243592 26758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.236324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33403 (local address 127.23.200.2:53391)
0806 20:37:50.237487 (+ 1163us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:50.237506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:50.237657 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:50.238021 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:50.238024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:50.238056 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:50.238523 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.238535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.239637 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.239640 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.241377 (+ 1737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.241395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.242764 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.242768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.242939 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.242975 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 20:37:50.243066 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 20:37:50.243183 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"mutex_wait_us":45,"spinlock_wait_cycles":768,"thread_start_us":676,"threads_started":1}
I20260806 20:37:50.244340 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.236256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:53391 (local address 127.23.200.1:33403)
0806 20:37:50.237049 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:50.237055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:50.237634 (+ 579us) server_negotiation.cc:407] Connection header received
0806 20:37:50.237780 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:50.237784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:50.237877 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:50.237984 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:50.238678 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.239497 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.241534 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.242511 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.242665 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.242737 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:50.243113 (+ 376us) server_negotiation.cc:299] Negotiation successful
0806 20:37:50.243267 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":515,"threads_started":1}
I20260806 20:37:50.244623 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58763c47817843468a21804e78bb20be" candidate_uuid: "c2a4a348653a4492a6bf999b51fd0806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707" is_pre_election: true
I20260806 20:37:50.244885 26669 raft_consensus.cc:2454] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a4a348653a4492a6bf999b51fd0806 in term 0.
I20260806 20:37:50.245116 26374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58763c47817843468a21804e78bb20be" candidate_uuid: "c2a4a348653a4492a6bf999b51fd0806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" is_pre_election: true
I20260806 20:37:50.245395 26450 leader_election.cc:302] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38eae1ef5d3742bbb6f431ee03191707, c2a4a348653a4492a6bf999b51fd0806; no voters:
I20260806 20:37:50.245563 26374 raft_consensus.cc:2454] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c2a4a348653a4492a6bf999b51fd0806 in term 0.
I20260806 20:37:50.245776 26741 raft_consensus.cc:2781] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:50.245877 26741 raft_consensus.cc:482] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:50.245936 26741 raft_consensus.cc:3037] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:50.248065 26741 raft_consensus.cc:504] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.248616 26741 leader_election.cc:288] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [CANDIDATE]: Term 1 election: Requested vote from peers 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:50.249051 26374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58763c47817843468a21804e78bb20be" candidate_uuid: "c2a4a348653a4492a6bf999b51fd0806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565"
I20260806 20:37:50.249274 26374 raft_consensus.cc:3037] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:50.249204 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58763c47817843468a21804e78bb20be" candidate_uuid: "c2a4a348653a4492a6bf999b51fd0806" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707"
I20260806 20:37:50.249451 26669 raft_consensus.cc:3037] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:50.251461 26669 raft_consensus.cc:2454] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a4a348653a4492a6bf999b51fd0806 in term 1.
I20260806 20:37:50.251506 26374 raft_consensus.cc:2454] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c2a4a348653a4492a6bf999b51fd0806 in term 1.
I20260806 20:37:50.251857 26450 leader_election.cc:302] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38eae1ef5d3742bbb6f431ee03191707, c2a4a348653a4492a6bf999b51fd0806; no voters:
I20260806 20:37:50.252159 26741 raft_consensus.cc:2781] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:50.252588 26741 raft_consensus.cc:686] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [term 1 LEADER]: Becoming Leader. State: Replica: c2a4a348653a4492a6bf999b51fd0806, State: Running, Role: LEADER
I20260806 20:37:50.253218 26741 consensus_queue.cc:237] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.256511 26760 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: TxnStatusTablet state changed. Reason: New leader c2a4a348653a4492a6bf999b51fd0806. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.256951 26760 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:37:50.257651 26762 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 20:37:50.257961 26762 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 20:37:50.258808 26202 catalog_manager.cc:5495] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 reported cstate change: term changed from 0 to 1, leader changed from <none> to c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2). New cstate: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } health_report { overall_health: UNKNOWN } } }
I20260806 20:37:50.259452 26742 consensus_queue.cc:1035] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:50.268733 26745 consensus_queue.cc:1035] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:50.334075 26753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.328177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35151 (local address 127.0.0.1:38610)
0806 20:37:50.328464 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:50.328480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:50.328654 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:50.329178 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:50.329183 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:50.329194 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:50.329401 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.329411 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.330593 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.330599 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.331603 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.331615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.332408 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.332414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.332587 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.332642 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.333503 (+ 861us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.333525 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:37:50.333614 (+ 89us) client_negotiation.cc:240] Negotiation successful
0806 20:37:50.333762 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 20:37:50.334087 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.328197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38610 (local address 127.23.200.2:35151)
0806 20:37:50.328395 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:50.328402 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:50.328559 (+ 157us) server_negotiation.cc:407] Connection header received
0806 20:37:50.328894 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:50.328904 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:50.329012 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:50.329156 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:50.329538 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.330467 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.331767 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.332226 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.332375 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.332859 (+ 484us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.333388 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.333552 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:50.333633 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 20:37:50.333719 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 20:37:50.341569 26669 raft_consensus.cc:1261] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Refusing update from remote peer c2a4a348653a4492a6bf999b51fd0806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:50.341843 26374 raft_consensus.cc:1261] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Refusing update from remote peer c2a4a348653a4492a6bf999b51fd0806: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:50.342290 26760 consensus_queue.cc:1035] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:50.342691 26741 consensus_queue.cc:1035] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:50.347536 26745 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: TxnStatusTablet state changed. Reason: New leader c2a4a348653a4492a6bf999b51fd0806. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.347803 26745 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:50.360728 26745 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.361083 26745 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P 38e88c4cf75a4ddeaffd27f1a5e8a565: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:50.364878 26740 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: TxnStatusTablet state changed. Reason: New leader c2a4a348653a4492a6bf999b51fd0806. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.364884 26760 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.365152 26740 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:50.365195 26760 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:37:50.366619 26741 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.366902 26741 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P c2a4a348653a4492a6bf999b51fd0806: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:37:50.370103 26740 tablet_replica.cc:439] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c2a4a348653a4492a6bf999b51fd0806" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } } }
I20260806 20:37:50.370361 26740 tablet_replica.cc:442] T 58763c47817843468a21804e78bb20be P 38eae1ef5d3742bbb6f431ee03191707: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:37:50.383325 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.376636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.23.200.1:33403)
0806 20:37:50.376845 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:50.376852 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:50.377499 (+ 647us) server_negotiation.cc:407] Connection header received
0806 20:37:50.377688 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:50.377695 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:50.377796 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:50.377963 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:50.378448 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.379577 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.380911 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.381542 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.381670 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.382032 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.382730 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.382890 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:50.382939 (+ 49us) server_negotiation.cc:299] Negotiation successful
0806 20:37:50.383024 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:37:50.402285 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.395073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:51143 (local address 127.23.200.2:35151)
0806 20:37:50.395241 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:50.395247 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:50.395827 (+ 580us) server_negotiation.cc:407] Connection header received
0806 20:37:50.396042 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:50.396048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:50.396159 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:50.396260 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:50.396850 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.397952 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.400131 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.400812 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.400953 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.401478 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.401605 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.401699 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:50.401909 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 20:37:50.402000 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 20:37:50.402400 26780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.395076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35151 (local address 127.23.200.1:51143)
0806 20:37:50.395712 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:50.395730 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:50.395869 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:50.396280 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:50.396285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:50.396298 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:50.396678 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.396689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.398090 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.398095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.399980 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.399995 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.401021 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.401027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.401194 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.401261 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.401744 (+ 483us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.401766 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:37:50.401857 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 20:37:50.402003 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":321,"threads_started":1}
I20260806 20:37:50.460101 26757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.442903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:39085 (local address 127.23.200.1:33403)
0806 20:37:50.443070 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:50.443076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:50.453727 (+ 10651us) server_negotiation.cc:407] Connection header received
0806 20:37:50.453954 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:50.453959 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:50.454028 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:50.454129 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:50.454726 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.455812 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.457754 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.458228 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.458410 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.458901 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.459006 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.459109 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:50.459302 (+ 193us) server_negotiation.cc:299] Negotiation successful
0806 20:37:50.459394 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 20:37:50.460577 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:50.442095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33403 (local address 127.23.200.2:39085)
0806 20:37:50.453603 (+ 11508us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:50.453620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:50.453794 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:50.454147 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:50.454151 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:50.454162 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:50.454565 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.454575 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.455992 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.455997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.457599 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:50.457612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:50.458429 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:50.458436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:50.458629 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:50.458691 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:50.459173 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:50.459183 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 20:37:50.459278 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 20:37:50.459419 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11292,"thread_start_us":11114,"threads_started":1}
I20260806 20:37:50.501909 26203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:50.502208 26203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:50.506497 26203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Sending DeleteTablet for 3 replicas of tablet 072b07f1fafd4ed38d7dae248fc9f75f
I20260806 20:37:50.507742 26351 tablet_service.cc:1514] Processing DeleteTablet for tablet 072b07f1fafd4ed38d7dae248fc9f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:50 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35074
I20260806 20:37:50.507769 26644 tablet_service.cc:1514] Processing DeleteTablet for tablet 072b07f1fafd4ed38d7dae248fc9f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:50 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:40474
I20260806 20:37:50.508854 26505 tablet_service.cc:1514] Processing DeleteTablet for tablet 072b07f1fafd4ed38d7dae248fc9f75f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:50 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 20:37:50.509933 26203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:37:50.511270 26203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:50.526126 26643 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 (DEFAULT_TABLE table=test-table [id=fe8e5ad6fb7a4f8cbb0409999d6a3ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:50.526479 26349 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 (DEFAULT_TABLE table=test-table [id=fe8e5ad6fb7a4f8cbb0409999d6a3ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:50.526751 26643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:50.526978 26349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:50.531213 26503 tablet_service.cc:1467] Processing CreateTablet for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 (DEFAULT_TABLE table=test-table [id=fe8e5ad6fb7a4f8cbb0409999d6a3ed1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:50.531929 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:50.540279 26794 tablet_bootstrap.cc:492] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Bootstrap starting.
I20260806 20:37:50.544000 26789 tablet_replica.cc:330] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: stopping tablet replica
I20260806 20:37:50.544580 26789 raft_consensus.cc:2229] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:37:50.545450 26789 raft_consensus.cc:2258] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:50.545641 26791 tablet_replica.cc:330] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: stopping tablet replica
I20260806 20:37:50.546295 26792 tablet_bootstrap.cc:492] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Bootstrap starting.
I20260806 20:37:50.546209 26791 raft_consensus.cc:2229] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:50.546607 26791 raft_consensus.cc:2258] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:50.548648 26790 tablet_replica.cc:330] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: stopping tablet replica
I20260806 20:37:50.546654 26794 tablet_bootstrap.cc:654] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:50.550760 26791 ts_tablet_manager.cc:1905] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:50.550802 26790 raft_consensus.cc:2229] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:50.551383 26790 raft_consensus.cc:2258] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:50.548146 26789 ts_tablet_manager.cc:1905] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:50.551980 26793 tablet_bootstrap.cc:492] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Bootstrap starting.
I20260806 20:37:50.559612 26792 tablet_bootstrap.cc:654] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:50.560037 26791 ts_tablet_manager.cc:1918] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:50.561388 26791 log.cc:1196] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/072b07f1fafd4ed38d7dae248fc9f75f
I20260806 20:37:50.561647 26790 ts_tablet_manager.cc:1905] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:50.562251 26791 ts_tablet_manager.cc:1939] T 072b07f1fafd4ed38d7dae248fc9f75f P c2a4a348653a4492a6bf999b51fd0806: Deleting consensus metadata
I20260806 20:37:50.564846 26191 catalog_manager.cc:4841] TS c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151): tablet 072b07f1fafd4ed38d7dae248fc9f75f (table test-table [id=e1a6b492b3294b2ea624ad60cdf184d4]) successfully deleted
I20260806 20:37:50.563697 26793 tablet_bootstrap.cc:654] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:50.572489 26793 tablet_bootstrap.cc:492] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: No bootstrap required, opened a new log
I20260806 20:37:50.576422 26793 ts_tablet_manager.cc:1397] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.000s
I20260806 20:37:50.579804 26792 tablet_bootstrap.cc:492] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: No bootstrap required, opened a new log
I20260806 20:37:50.579984 26792 ts_tablet_manager.cc:1397] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.000s
I20260806 20:37:50.580770 26794 tablet_bootstrap.cc:492] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: No bootstrap required, opened a new log
I20260806 20:37:50.581014 26794 ts_tablet_manager.cc:1397] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Time spent bootstrapping tablet: real 0.041s user 0.013s sys 0.004s
I20260806 20:37:50.580832 26792 raft_consensus.cc:348] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.581111 26792 raft_consensus.cc:374] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:50.581233 26792 raft_consensus.cc:729] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Initialized, Role: FOLLOWER
I20260806 20:37:50.581521 26792 consensus_queue.cc:260] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.582274 26792 ts_tablet_manager.cc:1428] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 20:37:50.582515 26793 raft_consensus.cc:348] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.582772 26793 raft_consensus.cc:374] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:50.582964 26793 raft_consensus.cc:729] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38eae1ef5d3742bbb6f431ee03191707, State: Initialized, Role: FOLLOWER
I20260806 20:37:50.583218 26793 consensus_queue.cc:260] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.583856 26793 ts_tablet_manager.cc:1428] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260806 20:37:50.587082 26790 ts_tablet_manager.cc:1918] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:50.588392 26794 raft_consensus.cc:348] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.588781 26794 raft_consensus.cc:374] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:50.588943 26794 raft_consensus.cc:729] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a4a348653a4492a6bf999b51fd0806, State: Initialized, Role: FOLLOWER
I20260806 20:37:50.589044 26790 log.cc:1196] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/072b07f1fafd4ed38d7dae248fc9f75f
I20260806 20:37:50.589335 26794 consensus_queue.cc:260] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.589689 26790 ts_tablet_manager.cc:1939] T 072b07f1fafd4ed38d7dae248fc9f75f P 38eae1ef5d3742bbb6f431ee03191707: Deleting consensus metadata
I20260806 20:37:50.591631 26192 catalog_manager.cc:4841] TS 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583): tablet 072b07f1fafd4ed38d7dae248fc9f75f (table test-table [id=e1a6b492b3294b2ea624ad60cdf184d4]) successfully deleted
I20260806 20:37:50.592206 26794 ts_tablet_manager.cc:1428] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20260806 20:37:50.592401 26789 ts_tablet_manager.cc:1918] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:50.592631 26789 log.cc:1196] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/072b07f1fafd4ed38d7dae248fc9f75f
I20260806 20:37:50.593400 26789 ts_tablet_manager.cc:1939] T 072b07f1fafd4ed38d7dae248fc9f75f P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting consensus metadata
I20260806 20:37:50.595141 26191 catalog_manager.cc:4841] TS 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403): tablet 072b07f1fafd4ed38d7dae248fc9f75f (table test-table [id=e1a6b492b3294b2ea624ad60cdf184d4]) successfully deleted
I20260806 20:37:50.617857 26782 raft_consensus.cc:482] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:50.618173 26782 raft_consensus.cc:504] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.619086 26782 leader_election.cc:288] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:50.619813 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3fcacaea14a1389ee0fe0b13e9e57" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707" is_pre_election: true
I20260806 20:37:50.619819 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3fcacaea14a1389ee0fe0b13e9e57" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806" is_pre_election: true
I20260806 20:37:50.620078 26669 raft_consensus.cc:2454] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 0.
I20260806 20:37:50.620090 26525 raft_consensus.cc:2454] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 0.
I20260806 20:37:50.620616 26301 leader_election.cc:302] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, 38eae1ef5d3742bbb6f431ee03191707; no voters:
I20260806 20:37:50.620893 26782 raft_consensus.cc:2781] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:50.621017 26782 raft_consensus.cc:482] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:50.621121 26782 raft_consensus.cc:3037] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:50.623893 26782 raft_consensus.cc:504] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.624512 26782 leader_election.cc:288] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 election: Requested vote from peers c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:50.624985 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3fcacaea14a1389ee0fe0b13e9e57" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806"
I20260806 20:37:50.625255 26525 raft_consensus.cc:3037] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:50.625419 26669 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1dc3fcacaea14a1389ee0fe0b13e9e57" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707"
I20260806 20:37:50.625648 26669 raft_consensus.cc:3037] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:50.628712 26525 raft_consensus.cc:2454] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 1.
I20260806 20:37:50.629184 26300 leader_election.cc:302] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, c2a4a348653a4492a6bf999b51fd0806; no voters:
I20260806 20:37:50.629345 26669 raft_consensus.cc:2454] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 1.
I20260806 20:37:50.629673 26782 raft_consensus.cc:2781] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:50.629863 26782 raft_consensus.cc:686] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 LEADER]: Becoming Leader. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Running, Role: LEADER
I20260806 20:37:50.630188 26782 consensus_queue.cc:237] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:50.633349 26202 catalog_manager.cc:5495] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } health_report { overall_health: UNKNOWN } } }
I20260806 20:37:50.662061 26669 raft_consensus.cc:1261] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Refusing update from remote peer 38e88c4cf75a4ddeaffd27f1a5e8a565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:50.662698 26745 consensus_queue.cc:1035] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:50.663049 26525 raft_consensus.cc:1261] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Refusing update from remote peer 38e88c4cf75a4ddeaffd27f1a5e8a565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:50.663841 26782 consensus_queue.cc:1035] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:50.668810 26768 mvcc.cc:205] Tried to move back new op lower bound from 7315655355024011264 to 7315655354903973888. Current Snapshot: MvccSnapshot[applied={T|T < 7315655355024011264}]
I20260806 20:37:50.669967 26765 mvcc.cc:205] Tried to move back new op lower bound from 7315655355024011264 to 7315655354903973888. Current Snapshot: MvccSnapshot[applied={T|T < 7315655355024011264}]
W20260806 20:37:50.693373 26765 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 20:37:52.304991 26202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:52.305217 26202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42668:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:52.309722 26202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Sending DeleteTablet for 3 replicas of tablet 1dc3fcacaea14a1389ee0fe0b13e9e57
I20260806 20:37:52.310593 26351 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:52 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35074
I20260806 20:37:52.310637 26503 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:52 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 20:37:52.310637 26643 tablet_service.cc:1514] Processing DeleteTablet for tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:52 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:40474
I20260806 20:37:52.323009 26825 tablet_replica.cc:330] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: stopping tablet replica
I20260806 20:37:52.323117 26826 tablet_replica.cc:330] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: stopping tablet replica
I20260806 20:37:52.323448 26825 raft_consensus.cc:2229] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:37:52.323794 26826 raft_consensus.cc:2229] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:52.324002 26825 raft_consensus.cc:2258] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:52.324342 26826 raft_consensus.cc:2258] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:52.325186 26827 tablet_replica.cc:330] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: stopping tablet replica
I20260806 20:37:52.325241 26825 ts_tablet_manager.cc:1905] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:52.325443 26827 raft_consensus.cc:2229] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:52.325459 26826 ts_tablet_manager.cc:1905] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:52.325718 26827 raft_consensus.cc:2258] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:52.331367 26827 ts_tablet_manager.cc:1905] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:52.333217 26825 ts_tablet_manager.cc:1918] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 20:37:52.333217 26826 ts_tablet_manager.cc:1918] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 20:37:52.333369 26826 log.cc:1196] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/1dc3fcacaea14a1389ee0fe0b13e9e57
I20260806 20:37:52.333549 26825 log.cc:1196] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/1dc3fcacaea14a1389ee0fe0b13e9e57
I20260806 20:37:52.334055 26826 ts_tablet_manager.cc:1939] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P c2a4a348653a4492a6bf999b51fd0806: Deleting consensus metadata
I20260806 20:37:52.334151 26825 ts_tablet_manager.cc:1939] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting consensus metadata
I20260806 20:37:52.335337 26191 catalog_manager.cc:4841] TS 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403): tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 (table test-table [id=fe8e5ad6fb7a4f8cbb0409999d6a3ed1]) successfully deleted
I20260806 20:37:52.335685 26191 catalog_manager.cc:4841] TS c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151): tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 (table test-table [id=fe8e5ad6fb7a4f8cbb0409999d6a3ed1]) successfully deleted
I20260806 20:37:52.337666 26827 ts_tablet_manager.cc:1918] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 20:37:52.337818 26827 log.cc:1196] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/1dc3fcacaea14a1389ee0fe0b13e9e57
I20260806 20:37:52.338228 26827 ts_tablet_manager.cc:1939] T 1dc3fcacaea14a1389ee0fe0b13e9e57 P 38eae1ef5d3742bbb6f431ee03191707: Deleting consensus metadata
I20260806 20:37:52.339221 26192 catalog_manager.cc:4841] TS 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583): tablet 1dc3fcacaea14a1389ee0fe0b13e9e57 (table test-table [id=fe8e5ad6fb7a4f8cbb0409999d6a3ed1]) successfully deleted
Aug 06 20:37:52 dist-test-slave-tm1g krb5kdc[26159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 20:37:52 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:52.378193 26835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.357241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42692 (local address 127.23.200.62:45099)
0806 20:37:52.358122 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.358129 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.358383 (+ 254us) server_negotiation.cc:407] Connection header received
0806 20:37:52.358611 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.358615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.358680 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.358795 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:52.360462 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.361264 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.362808 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.363339 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.363491 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.372727 (+ 9236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:52.372733 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:52.372736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:52.372773 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:52.375569 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:52.376257 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:52.376261 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:52.376263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:52.376410 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:52.376766 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:52.376769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:52.376771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:52.377130 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:52.377312 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.377555 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.377802 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":362,"threads_started":1}
I20260806 20:37:52.381575 26202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:37:52.382490 26202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:52.389052 26503 tablet_service.cc:1467] Processing CreateTablet for tablet 7672cc9a5c5742c88462dde70f51be00 (DEFAULT_TABLE table=test-table [id=f9c5e66664e14526b266a4f7c7095fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:52.389379 26351 tablet_service.cc:1467] Processing CreateTablet for tablet 7672cc9a5c5742c88462dde70f51be00 (DEFAULT_TABLE table=test-table [id=f9c5e66664e14526b266a4f7c7095fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:52.389592 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672cc9a5c5742c88462dde70f51be00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:52.389864 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672cc9a5c5742c88462dde70f51be00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:52.395227 26643 tablet_service.cc:1467] Processing CreateTablet for tablet 7672cc9a5c5742c88462dde70f51be00 (DEFAULT_TABLE table=test-table [id=f9c5e66664e14526b266a4f7c7095fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:52.395629 26643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7672cc9a5c5742c88462dde70f51be00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:52.405588 26837 tablet_bootstrap.cc:492] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Bootstrap starting.
I20260806 20:37:52.405692 26838 tablet_bootstrap.cc:492] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Bootstrap starting.
I20260806 20:37:52.406824 26836 tablet_bootstrap.cc:492] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Bootstrap starting.
I20260806 20:37:52.408311 26838 tablet_bootstrap.cc:654] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:52.408344 26837 tablet_bootstrap.cc:654] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:52.408741 26836 tablet_bootstrap.cc:654] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:52.410924 26837 tablet_bootstrap.cc:492] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: No bootstrap required, opened a new log
I20260806 20:37:52.410960 26838 tablet_bootstrap.cc:492] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: No bootstrap required, opened a new log
I20260806 20:37:52.411094 26838 ts_tablet_manager.cc:1397] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 20:37:52.411087 26837 ts_tablet_manager.cc:1397] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 20:37:52.411945 26837 raft_consensus.cc:348] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.411945 26838 raft_consensus.cc:348] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.412186 26838 raft_consensus.cc:374] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:52.412186 26837 raft_consensus.cc:374] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:52.412287 26837 raft_consensus.cc:729] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a4a348653a4492a6bf999b51fd0806, State: Initialized, Role: FOLLOWER
I20260806 20:37:52.412333 26838 raft_consensus.cc:729] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38eae1ef5d3742bbb6f431ee03191707, State: Initialized, Role: FOLLOWER
I20260806 20:37:52.412556 26837 consensus_queue.cc:260] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.412621 26838 consensus_queue.cc:260] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.413292 26837 ts_tablet_manager.cc:1428] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 20:37:52.419101 26838 ts_tablet_manager.cc:1428] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260806 20:37:52.421556 26836 tablet_bootstrap.cc:492] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: No bootstrap required, opened a new log
I20260806 20:37:52.421715 26836 ts_tablet_manager.cc:1397] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20260806 20:37:52.422699 26836 raft_consensus.cc:348] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.422955 26836 raft_consensus.cc:374] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:52.423075 26836 raft_consensus.cc:729] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Initialized, Role: FOLLOWER
I20260806 20:37:52.423339 26836 consensus_queue.cc:260] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.423834 26836 ts_tablet_manager.cc:1428] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260806 20:37:52.491206 26820 raft_consensus.cc:482] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:52.491452 26820 raft_consensus.cc:504] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.492316 26820 leader_election.cc:288] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:52.492905 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672cc9a5c5742c88462dde70f51be00" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806" is_pre_election: true
I20260806 20:37:52.493088 26668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672cc9a5c5742c88462dde70f51be00" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707" is_pre_election: true
I20260806 20:37:52.493257 26525 raft_consensus.cc:2454] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 0.
I20260806 20:37:52.493322 26668 raft_consensus.cc:2454] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 0.
I20260806 20:37:52.493798 26301 leader_election.cc:302] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, 38eae1ef5d3742bbb6f431ee03191707, c2a4a348653a4492a6bf999b51fd0806; no voters:
I20260806 20:37:52.494108 26820 raft_consensus.cc:2781] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:52.494256 26820 raft_consensus.cc:482] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:52.494410 26820 raft_consensus.cc:3037] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:52.496811 26820 raft_consensus.cc:504] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.497923 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672cc9a5c5742c88462dde70f51be00" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806"
I20260806 20:37:52.498227 26525 raft_consensus.cc:3037] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:52.498339 26668 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7672cc9a5c5742c88462dde70f51be00" candidate_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38eae1ef5d3742bbb6f431ee03191707"
I20260806 20:37:52.498584 26668 raft_consensus.cc:3037] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:52.498668 26820 leader_election.cc:288] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 election: Requested vote from peers c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151), 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583)
I20260806 20:37:52.501695 26525 raft_consensus.cc:2454] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 1.
I20260806 20:37:52.501708 26668 raft_consensus.cc:2454] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38e88c4cf75a4ddeaffd27f1a5e8a565 in term 1.
I20260806 20:37:52.502135 26300 leader_election.cc:302] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, c2a4a348653a4492a6bf999b51fd0806; no voters:
I20260806 20:37:52.502516 26820 raft_consensus.cc:2781] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:52.502701 26820 raft_consensus.cc:686] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 LEADER]: Becoming Leader. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Running, Role: LEADER
I20260806 20:37:52.503012 26820 consensus_queue.cc:237] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.506682 26202 catalog_manager.cc:5495] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } health_report { overall_health: UNKNOWN } } }
I20260806 20:37:52.545475 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.537949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35082 (local address 127.23.200.1:33403)
0806 20:37:52.538871 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.538879 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.538936 (+ 57us) server_negotiation.cc:407] Connection header received
0806 20:37:52.539015 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.539023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.539124 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.539243 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:52.539563 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.541080 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.542904 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.543752 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.543919 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.544359 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:52.544532 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:52.544633 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.544781 (+ 148us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.544889 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":500,"threads_started":1}
I20260806 20:37:52.556144 26525 raft_consensus.cc:1261] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Refusing update from remote peer 38e88c4cf75a4ddeaffd27f1a5e8a565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:52.556165 26668 raft_consensus.cc:1261] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Refusing update from remote peer 38e88c4cf75a4ddeaffd27f1a5e8a565: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:52.556885 26820 consensus_queue.cc:1035] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:52.557312 26817 consensus_queue.cc:1035] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:52.581341 26802 mvcc.cc:205] Tried to move back new op lower bound from 7315655362784759808 to 7315655362574749696. Current Snapshot: MvccSnapshot[applied={T|T < 7315655362784759808}]
I20260806 20:37:52.618221 26203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:52.618710 26203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:52.621991 26203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Sending DeleteTablet for 3 replicas of tablet 7672cc9a5c5742c88462dde70f51be00
I20260806 20:37:52.622946 26503 tablet_service.cc:1514] Processing DeleteTablet for tablet 7672cc9a5c5742c88462dde70f51be00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:52 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 20:37:52.622948 26351 tablet_service.cc:1514] Processing DeleteTablet for tablet 7672cc9a5c5742c88462dde70f51be00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:52 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35074
I20260806 20:37:52.623155 26643 tablet_service.cc:1514] Processing DeleteTablet for tablet 7672cc9a5c5742c88462dde70f51be00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:52 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:40474
I20260806 20:37:52.623248 26825 tablet_replica.cc:330] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: stopping tablet replica
I20260806 20:37:52.623260 26826 tablet_replica.cc:330] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: stopping tablet replica
I20260806 20:37:52.623435 26827 tablet_replica.cc:330] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: stopping tablet replica
I20260806 20:37:52.623458 26825 raft_consensus.cc:2229] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:37:52.623463 26826 raft_consensus.cc:2229] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:52.623643 26827 raft_consensus.cc:2229] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:52.623777 26826 raft_consensus.cc:2258] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:52.623804 26825 raft_consensus.cc:2258] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:52.623930 26827 raft_consensus.cc:2258] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:52.624866 26203 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 20:37:52.625344 26825 ts_tablet_manager.cc:1905] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:52.625372 26827 ts_tablet_manager.cc:1905] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260806 20:37:52.625661 26203 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:37:52.630823 26825 ts_tablet_manager.cc:1918] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:52.631146 26825 log.cc:1196] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/7672cc9a5c5742c88462dde70f51be00
I20260806 20:37:52.631544 26826 ts_tablet_manager.cc:1905] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:52.631783 26825 ts_tablet_manager.cc:1939] T 7672cc9a5c5742c88462dde70f51be00 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting consensus metadata
I20260806 20:37:52.634517 26191 catalog_manager.cc:4841] TS 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403): tablet 7672cc9a5c5742c88462dde70f51be00 (table test-table [id=f9c5e66664e14526b266a4f7c7095fec]) successfully deleted
I20260806 20:37:52.636759 26643 tablet_service.cc:1467] Processing CreateTablet for tablet a32d9097b9e84a6d9f1b97ca0fd5b666 (DEFAULT_TABLE table=test-table [id=5e8a8c72ef9e4b689eb2e2e8ddbde9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:52.637192 26643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32d9097b9e84a6d9f1b97ca0fd5b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:52.638270 26827 ts_tablet_manager.cc:1918] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:52.638475 26827 log.cc:1196] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/7672cc9a5c5742c88462dde70f51be00
I20260806 20:37:52.639067 26827 ts_tablet_manager.cc:1939] T 7672cc9a5c5742c88462dde70f51be00 P 38eae1ef5d3742bbb6f431ee03191707: Deleting consensus metadata
I20260806 20:37:52.640775 26351 tablet_service.cc:1467] Processing CreateTablet for tablet a32d9097b9e84a6d9f1b97ca0fd5b666 (DEFAULT_TABLE table=test-table [id=5e8a8c72ef9e4b689eb2e2e8ddbde9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:52.641028 26503 tablet_service.cc:1467] Processing CreateTablet for tablet a32d9097b9e84a6d9f1b97ca0fd5b666 (DEFAULT_TABLE table=test-table [id=5e8a8c72ef9e4b689eb2e2e8ddbde9ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:37:52.641273 26351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32d9097b9e84a6d9f1b97ca0fd5b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:52.641470 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a32d9097b9e84a6d9f1b97ca0fd5b666. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:52.645741 26838 tablet_bootstrap.cc:492] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Bootstrap starting.
I20260806 20:37:52.647939 26838 tablet_bootstrap.cc:654] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:52.650732 26836 tablet_bootstrap.cc:492] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Bootstrap starting.
I20260806 20:37:52.650836 26826 ts_tablet_manager.cc:1918] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:37:52.651046 26826 log.cc:1196] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/7672cc9a5c5742c88462dde70f51be00
I20260806 20:37:52.651736 26826 ts_tablet_manager.cc:1939] T 7672cc9a5c5742c88462dde70f51be00 P c2a4a348653a4492a6bf999b51fd0806: Deleting consensus metadata
I20260806 20:37:52.652064 26192 catalog_manager.cc:4841] TS 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583): tablet 7672cc9a5c5742c88462dde70f51be00 (table test-table [id=f9c5e66664e14526b266a4f7c7095fec]) successfully deleted
I20260806 20:37:52.652840 26836 tablet_bootstrap.cc:654] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:52.653407 26191 catalog_manager.cc:4841] TS c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151): tablet 7672cc9a5c5742c88462dde70f51be00 (table test-table [id=f9c5e66664e14526b266a4f7c7095fec]) successfully deleted
I20260806 20:37:52.653980 26837 tablet_bootstrap.cc:492] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Bootstrap starting.
I20260806 20:37:52.655838 26838 tablet_bootstrap.cc:492] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: No bootstrap required, opened a new log
I20260806 20:37:52.656016 26838 ts_tablet_manager.cc:1397] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.006s
I20260806 20:37:52.656219 26837 tablet_bootstrap.cc:654] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:52.658533 26838 raft_consensus.cc:348] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.658874 26838 raft_consensus.cc:374] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:52.658963 26838 raft_consensus.cc:729] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38eae1ef5d3742bbb6f431ee03191707, State: Initialized, Role: FOLLOWER
I20260806 20:37:52.659391 26838 consensus_queue.cc:260] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.664501 26838 ts_tablet_manager.cc:1428] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260806 20:37:52.667578 26836 tablet_bootstrap.cc:492] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: No bootstrap required, opened a new log
I20260806 20:37:52.667807 26836 ts_tablet_manager.cc:1397] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20260806 20:37:52.667985 26837 tablet_bootstrap.cc:492] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: No bootstrap required, opened a new log
I20260806 20:37:52.668232 26837 ts_tablet_manager.cc:1397] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
I20260806 20:37:52.668874 26836 raft_consensus.cc:348] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.669210 26836 raft_consensus.cc:374] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:52.669438 26836 raft_consensus.cc:729] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e88c4cf75a4ddeaffd27f1a5e8a565, State: Initialized, Role: FOLLOWER
I20260806 20:37:52.669420 26837 raft_consensus.cc:348] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.669800 26837 raft_consensus.cc:374] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:52.669955 26837 raft_consensus.cc:729] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2a4a348653a4492a6bf999b51fd0806, State: Initialized, Role: FOLLOWER
I20260806 20:37:52.669837 26836 consensus_queue.cc:260] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.670233 26837 consensus_queue.cc:260] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.670588 26836 ts_tablet_manager.cc:1428] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260806 20:37:52.671633 26837 ts_tablet_manager.cc:1428] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 20:37:52.701803 26842 raft_consensus.cc:482] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:37:52.702157 26842 raft_consensus.cc:504] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.704095 26842 leader_election.cc:288] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403), c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151)
I20260806 20:37:52.717114 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.704533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:50413 (local address 127.23.200.1:33403)
0806 20:37:52.704804 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.704813 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.706130 (+ 1317us) server_negotiation.cc:407] Connection header received
0806 20:37:52.706534 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.706540 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.706634 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.706707 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:52.707822 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.709143 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.712387 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.714235 (+ 1848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.714422 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.714534 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.716557 (+ 2023us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.716744 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 20:37:52.718041 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.704466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33403 (local address 127.23.200.3:50413)
0806 20:37:52.706015 (+ 1549us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:52.706038 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:52.706343 (+ 305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:52.707026 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:52.707034 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:52.707069 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:52.707607 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.707622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.709358 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.709366 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.712191 (+ 2825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.712211 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.716083 (+ 3872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.716091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.716329 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.716380 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 20:37:52.717397 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0806 20:37:52.717592 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1259,"thread_start_us":881,"threads_started":1}
I20260806 20:37:52.720197 26374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32d9097b9e84a6d9f1b97ca0fd5b666" candidate_uuid: "38eae1ef5d3742bbb6f431ee03191707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" is_pre_election: true
I20260806 20:37:52.720611 26374 raft_consensus.cc:2454] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38eae1ef5d3742bbb6f431ee03191707 in term 0.
I20260806 20:37:52.721233 26594 leader_election.cc:302] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, 38eae1ef5d3742bbb6f431ee03191707; no voters:
I20260806 20:37:52.721781 26842 raft_consensus.cc:2781] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:37:52.722033 26842 raft_consensus.cc:482] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:37:52.722180 26842 raft_consensus.cc:3037] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:52.725145 26842 raft_consensus.cc:504] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.725868 26842 leader_election.cc:288] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [CANDIDATE]: Term 1 election: Requested vote from peers 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403), c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151)
I20260806 20:37:52.726537 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.705773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35151 (local address 127.23.200.3:34381)
0806 20:37:52.718433 (+ 12660us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:52.718455 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:52.718572 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:52.719644 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:52.719649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:52.719678 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:37:52.719907 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.719928 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.721320 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.721328 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.723614 (+ 2286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.723628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.725663 (+ 2035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.725668 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.725944 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.725982 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:37:52.726082 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 20:37:52.726229 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12504,"thread_start_us":13562,"threads_started":1}
I20260806 20:37:52.726500 26374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32d9097b9e84a6d9f1b97ca0fd5b666" candidate_uuid: "38eae1ef5d3742bbb6f431ee03191707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565"
I20260806 20:37:52.726733 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.705888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:34381 (local address 127.23.200.2:35151)
0806 20:37:52.719270 (+ 13382us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.719278 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.719308 (+ 30us) server_negotiation.cc:407] Connection header received
0806 20:37:52.719387 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.719395 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.719483 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.719602 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:37:52.720133 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.721189 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.723783 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.725491 (+ 1708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.725690 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.725796 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.726114 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.726296 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13161,"thread_start_us":12941,"threads_started":1}
I20260806 20:37:52.726743 26374 raft_consensus.cc:3037] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:52.727662 26525 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32d9097b9e84a6d9f1b97ca0fd5b666" candidate_uuid: "38eae1ef5d3742bbb6f431ee03191707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806" is_pre_election: true
I20260806 20:37:52.727944 26525 raft_consensus.cc:2454] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38eae1ef5d3742bbb6f431ee03191707 in term 0.
I20260806 20:37:52.728451 26524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a32d9097b9e84a6d9f1b97ca0fd5b666" candidate_uuid: "38eae1ef5d3742bbb6f431ee03191707" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2a4a348653a4492a6bf999b51fd0806"
I20260806 20:37:52.728672 26524 raft_consensus.cc:3037] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:52.729956 26374 raft_consensus.cc:2454] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38eae1ef5d3742bbb6f431ee03191707 in term 1.
I20260806 20:37:52.730400 26594 leader_election.cc:302] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38e88c4cf75a4ddeaffd27f1a5e8a565, 38eae1ef5d3742bbb6f431ee03191707; no voters:
I20260806 20:37:52.730667 26842 raft_consensus.cc:2781] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:52.732214 26524 raft_consensus.cc:2454] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38eae1ef5d3742bbb6f431ee03191707 in term 1.
I20260806 20:37:52.733306 26842 raft_consensus.cc:686] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 LEADER]: Becoming Leader. State: Replica: 38eae1ef5d3742bbb6f431ee03191707, State: Running, Role: LEADER
I20260806 20:37:52.733908 26842 consensus_queue.cc:237] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } }
I20260806 20:37:52.737864 26202 catalog_manager.cc:5495] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3). New cstate: current_term: 1 leader_uuid: "38eae1ef5d3742bbb6f431ee03191707" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38eae1ef5d3742bbb6f431ee03191707" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37583 } health_report { overall_health: HEALTHY } } }
I20260806 20:37:52.772431 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.765365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.23.200.3:37583)
0806 20:37:52.766270 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.766274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.766294 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:37:52.766458 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.766462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.766546 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.766776 (+ 230us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:52.767220 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.768137 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.769705 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.770152 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.770247 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.770842 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:52.771727 (+ 885us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:52.771848 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.771976 (+ 128us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.772040 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":800,"thread_start_us":555,"threads_started":1}
I20260806 20:37:52.788153 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.782461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:44819 (local address 127.23.200.2:35151)
0806 20:37:52.782630 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.782635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.783359 (+ 724us) server_negotiation.cc:407] Connection header received
0806 20:37:52.783599 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.783604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.783669 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.783764 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:52.784240 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.785029 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.786506 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.786903 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.787040 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.787452 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:52.787544 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:52.787641 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.787775 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.787853 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:37:52.788264 26862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.782447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35151 (local address 127.23.200.3:44819)
0806 20:37:52.783241 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:52.783256 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:52.783436 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:52.783786 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:52.783790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:52.783801 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:52.784102 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.784109 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.785168 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.785174 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.786369 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.786381 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.787085 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.787094 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.787225 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.787273 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:52.787643 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:52.787657 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 20:37:52.787751 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 20:37:52.787887 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":418,"threads_started":1}
I20260806 20:37:52.796027 26524 raft_consensus.cc:1261] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Refusing update from remote peer 38eae1ef5d3742bbb6f431ee03191707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:52.796581 26374 raft_consensus.cc:1261] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Refusing update from remote peer 38eae1ef5d3742bbb6f431ee03191707: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:37:52.796687 26857 consensus_queue.cc:1035] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2a4a348653a4492a6bf999b51fd0806" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:52.797287 26842 consensus_queue.cc:1035] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e88c4cf75a4ddeaffd27f1a5e8a565" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33403 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:37:52.829851 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.821469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37583 (local address 127.23.200.2:42183)
0806 20:37:52.822766 (+ 1297us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:52.822831 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:52.822979 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:52.823343 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:52.823349 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:52.823362 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:37:52.823659 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.823670 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.825266 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.825272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.827255 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:52.827270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.828480 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.828488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:52.828690 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.828751 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:37:52.829190 (+ 439us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:37:52.829204 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 20:37:52.829299 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 20:37:52.829449 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":1055,"threads_started":1}
I20260806 20:37:52.829828 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:52.821898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:42183 (local address 127.23.200.3:37583)
0806 20:37:52.822073 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:52.822077 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:52.822902 (+ 825us) server_negotiation.cc:407] Connection header received
0806 20:37:52.823126 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:52.823132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:52.823211 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:52.823300 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:37:52.823852 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.825107 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.827409 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:52.828312 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:52.828444 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:52.828918 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:37:52.829079 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:37:52.829175 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:52.829334 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 20:37:52.829418 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 20:37:52.831324 26806 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 20:37:54.440900 26203 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:54.441212 26203 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:37:54.448362 26203 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f62a014d91a439c87517d0d27f789ed: Sending DeleteTablet for 3 replicas of tablet a32d9097b9e84a6d9f1b97ca0fd5b666
I20260806 20:37:54.449182 26351 tablet_service.cc:1514] Processing DeleteTablet for tablet a32d9097b9e84a6d9f1b97ca0fd5b666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:54 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35074
I20260806 20:37:54.449183 26503 tablet_service.cc:1514] Processing DeleteTablet for tablet a32d9097b9e84a6d9f1b97ca0fd5b666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:54 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38608
I20260806 20:37:54.449316 26643 tablet_service.cc:1514] Processing DeleteTablet for tablet a32d9097b9e84a6d9f1b97ca0fd5b666 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:37:54 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:40474
I20260806 20:37:54.450027 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26281
I20260806 20:37:54.469849 26894 tablet_replica.cc:330] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: stopping tablet replica
I20260806 20:37:54.470378 26894 raft_consensus.cc:2229] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:54.470830 26894 raft_consensus.cc:2258] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:54.472307 26894 ts_tablet_manager.cc:1905] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:54.487058 26895 tablet_replica.cc:330] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: stopping tablet replica
I20260806 20:37:54.487439 26895 raft_consensus.cc:2229] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:37:54.487815 26895 raft_consensus.cc:2258] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:54.489056 26895 ts_tablet_manager.cc:1905] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:54.490273 26894 ts_tablet_manager.cc:1918] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 20:37:54.490535 26894 log.cc:1196] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/a32d9097b9e84a6d9f1b97ca0fd5b666
I20260806 20:37:54.491133 26896 tablet_replica.cc:330] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: stopping tablet replica
I20260806 20:37:54.491142 26894 ts_tablet_manager.cc:1939] T a32d9097b9e84a6d9f1b97ca0fd5b666 P c2a4a348653a4492a6bf999b51fd0806: Deleting consensus metadata
Aug 06 20:37:54 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:37:54.491364 26896 raft_consensus.cc:2229] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:37:54.492054 26896 raft_consensus.cc:2258] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:37:54.492370 26191 catalog_manager.cc:4841] TS c2a4a348653a4492a6bf999b51fd0806 (127.23.200.2:35151): tablet a32d9097b9e84a6d9f1b97ca0fd5b666 (table test-table [id=5e8a8c72ef9e4b689eb2e2e8ddbde9ff]) successfully deleted
I20260806 20:37:54.493057 26896 ts_tablet_manager.cc:1905] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:37:54.497509 26895 ts_tablet_manager.cc:1918] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 20:37:54.497671 26895 log.cc:1196] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/a32d9097b9e84a6d9f1b97ca0fd5b666
I20260806 20:37:54.498276 26895 ts_tablet_manager.cc:1939] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38e88c4cf75a4ddeaffd27f1a5e8a565: Deleting consensus metadata
I20260806 20:37:54.498588 26896 ts_tablet_manager.cc:1918] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 20:37:54.498723 26896 log.cc:1196] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/a32d9097b9e84a6d9f1b97ca0fd5b666
I20260806 20:37:54.499135 26896 ts_tablet_manager.cc:1939] T a32d9097b9e84a6d9f1b97ca0fd5b666 P 38eae1ef5d3742bbb6f431ee03191707: Deleting consensus metadata
I20260806 20:37:54.499440 26191 catalog_manager.cc:4841] TS 38e88c4cf75a4ddeaffd27f1a5e8a565 (127.23.200.1:33403): tablet a32d9097b9e84a6d9f1b97ca0fd5b666 (table test-table [id=5e8a8c72ef9e4b689eb2e2e8ddbde9ff]) successfully deleted
I20260806 20:37:54.500169 26192 catalog_manager.cc:4841] TS 38eae1ef5d3742bbb6f431ee03191707 (127.23.200.3:37583): tablet a32d9097b9e84a6d9f1b97ca0fd5b666 (table test-table [id=5e8a8c72ef9e4b689eb2e2e8ddbde9ff]) successfully deleted
I20260806 20:37:54.501691 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:54.453119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.23.200.1:33403)
0806 20:37:54.477957 (+ 24838us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:54.477966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:54.477998 (+ 32us) server_negotiation.cc:407] Connection header received
0806 20:37:54.478078 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:54.478086 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:54.478196 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:54.480497 (+ 2301us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:54.480732 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:54.482404 (+ 1672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:54.485010 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:54.486112 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:54.486268 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:54.493162 (+ 6894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:54.493181 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:54.493202 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:54.493246 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:54.498360 (+ 5114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:54.499404 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:54.499411 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:54.499413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:54.499538 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:54.499973 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:54.499978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:54.499981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:54.500470 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:54.500816 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:54.501079 (+ 263us) server_negotiation.cc:299] Negotiation successful
0806 20:37:54.501318 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24654,"thread_start_us":29419,"threads_started":1}
W20260806 20:37:54.502357 26420 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35090
W20260806 20:37:54.503047 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26281: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:37:54.503309 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 26281
I20260806 20:37:54.529026 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26430
Aug 06 20:37:54 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:37:54.544277 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:54.529824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38622 (local address 127.23.200.2:35151)
0806 20:37:54.530380 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:54.530385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:54.530401 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:37:54.530454 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:54.530458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:54.530516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:54.530617 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:54.531866 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:54.532626 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:54.533853 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:54.534244 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:54.534427 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:54.538606 (+ 4179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:54.538617 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:54.538632 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:54.538664 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:54.541467 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:54.542134 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:54.542141 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:54.542144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:54.542271 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:54.542753 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:54.542759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:54.542762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:54.543109 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:54.543448 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:54.543696 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 20:37:54.543920 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":247,"threads_started":1}
W20260806 20:37:54.544999 26567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38622
W20260806 20:37:54.545678 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26430: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:37:54.545862 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 26430
I20260806 20:37:54.569530 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26576
Aug 06 20:37:54 dist-test-slave-tm1g krb5kdc[26159](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:37:54.584721 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:54.570348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40482 (local address 127.23.200.3:37583)
0806 20:37:54.570887 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:54.570891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:54.570907 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:37:54.570960 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:54.570964 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:54.571024 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:54.571146 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:54.572561 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:54.573327 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:54.574638 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:54.575080 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:54.575218 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:54.579499 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:54.579514 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:54.579531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:54.579570 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:54.582228 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:54.582976 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:54.582983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:54.582985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:54.583133 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:54.583475 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:54.583479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:54.583481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:54.583805 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:54.584031 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:54.584278 (+ 247us) server_negotiation.cc:299] Negotiation successful
0806 20:37:54.584460 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":272,"threads_started":1}
W20260806 20:37:54.585395 26709 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40482
W20260806 20:37:54.586018 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26576: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:37:54.586145 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 26576
I20260806 20:37:54.609709 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26173
I20260806 20:37:54.610939 26249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:54.730195 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 26173
2026-08-06T20:37:54Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8631 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:37:54 dist-test-slave-tm1g krb5kdc[26905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:37:54 dist-test-slave-tm1g krb5kdc[26905](info): set up 2 sockets
Aug 06 20:37:54 dist-test-slave-tm1g krb5kdc[26905](info): commencing operation
krb5kdc: starting...
W20260806 20:37:55.842861 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:37:55 dist-test-slave-tm1g krb5kdc[26905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:37:56.037585 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:38393
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:38393
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:56.362327 26918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:56.362682 26918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:56.362843 26918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:56.370040 26918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:37:56.370182 26918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:56.370311 26918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:56.370412 26918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:37:56.370507 26918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:37:56.380091 26918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:38393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:56.381390 26918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:56.382737 26918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:56.395035 26926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:56.395087 26924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:56.395107 26923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:56.395851 26918 server_base.cc:1034] running on GCE node
I20260806 20:37:56.396555 26918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:56.397801 26918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:56.399005 26918 hybrid_clock.cc:648] HybridClock initialized: now 1786048676398939 us; error 43 us; skew 500 ppm
I20260806 20:37:56.399295 26918 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:56 dist-test-slave-tm1g krb5kdc[26905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048676, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:56.402372 26918 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:37:56.403733 26918 webserver.cc:460] Webserver started at http://127.23.200.62:44755/ using document root <none> and password file <none>
I20260806 20:37:56.404440 26918 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:56.404573 26918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:56.404910 26918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:56.407877 26918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "99ded0d7f1d14cc5b61f26589812e9f0"
format_stamp: "Formatted at 2026-08-06 20:37:56 on dist-test-slave-tm1g"
server_key: "173acf0d5a14128e3f76f1c38f385bbd"
server_key_iv: "e570caf38ad079e3108f2367491deb1a"
server_key_version: "encryptionkey@0"
I20260806 20:37:56.408659 26918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "99ded0d7f1d14cc5b61f26589812e9f0"
format_stamp: "Formatted at 2026-08-06 20:37:56 on dist-test-slave-tm1g"
server_key: "173acf0d5a14128e3f76f1c38f385bbd"
server_key_iv: "e570caf38ad079e3108f2367491deb1a"
server_key_version: "encryptionkey@0"
I20260806 20:37:56.413645 26918 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:37:56.417026 26933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:56.418345 26918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:37:56.418540 26918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "99ded0d7f1d14cc5b61f26589812e9f0"
format_stamp: "Formatted at 2026-08-06 20:37:56 on dist-test-slave-tm1g"
server_key: "173acf0d5a14128e3f76f1c38f385bbd"
server_key_iv: "e570caf38ad079e3108f2367491deb1a"
server_key_version: "encryptionkey@0"
I20260806 20:37:56.418748 26918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:56.435632 26918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:56.439538 26918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:56.439823 26918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:56.458570 26918 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:38393
I20260806 20:37:56.458591 26985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:38393 every 8 connection(s)
I20260806 20:37:56.459913 26918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:37:56.463502 26986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:37:56.466462 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 26918
I20260806 20:37:56.466684 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:37:56.467473 24352 external_mini_cluster.cc:1428] Setting key 3d10e527703e38a4155cdbe9a5127197
I20260806 20:37:56.476189 26986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0: Bootstrap starting.
I20260806 20:37:56.480015 26986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0: Neither blocks nor log segments found. Creating new log.
I20260806 20:37:56.481364 26986 log.cc:824] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0: Log is configured to *not* fsync() on all Append() calls
Aug 06 20:37:56 dist-test-slave-tm1g krb5kdc[26905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:56.484429 26986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0: No bootstrap required, opened a new log
I20260806 20:37:56.492733 26989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:56.469629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.23.200.62:38393)
0806 20:37:56.470450 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:56.470462 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:56.470502 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:37:56.472172 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:56.472194 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:56.472590 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:56.472905 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:56.474750 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:56.476123 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:56.477926 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:56.478658 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:56.478785 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:56.483820 (+ 5035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:56.483831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:56.483847 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:56.483897 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:56.488022 (+ 4125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:56.488744 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:56.488753 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:56.488760 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:56.488982 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:56.489366 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:56.489372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:56.489375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:56.490097 (+ 722us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:56.490436 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:56.490873 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 20:37:56.491331 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":375,"threads_started":1}
I20260806 20:37:56.495147 26986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38393 } }
I20260806 20:37:56.495579 26986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:37:56.495725 26986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99ded0d7f1d14cc5b61f26589812e9f0, State: Initialized, Role: FOLLOWER
I20260806 20:37:56.496327 26986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38393 } }
I20260806 20:37:56.496543 26986 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:37:56.496641 26986 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:37:56.496755 26986 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:37:56.499193 26986 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38393 } }
I20260806 20:37:56.499686 26986 leader_election.cc:302] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99ded0d7f1d14cc5b61f26589812e9f0; no voters:
I20260806 20:37:56.500236 26986 leader_election.cc:288] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:37:56.500378 26991 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:37:56.501091 26991 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [term 1 LEADER]: Becoming Leader. State: Replica: 99ded0d7f1d14cc5b61f26589812e9f0, State: Running, Role: LEADER
I20260806 20:37:56.501631 26991 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38393 } }
I20260806 20:37:56.502210 26986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:37:56.505160 26992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38393 } } }
I20260806 20:37:56.505640 26992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [sys.catalog]: This master's current role is: LEADER
I20260806 20:37:56.509711 26993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99ded0d7f1d14cc5b61f26589812e9f0. Latest consensus state: current_term: 1 leader_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99ded0d7f1d14cc5b61f26589812e9f0" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38393 } } }
I20260806 20:37:56.510144 26993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0 [sys.catalog]: This master's current role is: LEADER
W20260806 20:37:56.513777 27008 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:37:56.513877 27008 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:37:56.514032 27000 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:37:56.515717 27000 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:37:56.523479 27000 catalog_manager.cc:1347] Generated new cluster ID: ac3f72589ec84182ad7f983d9f295018
I20260806 20:37:56.523583 27000 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:37:56.540334 27000 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:37:56.541616 27000 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:37:56.549126 27000 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99ded0d7f1d14cc5b61f26589812e9f0: Generated new TSK 0
I20260806 20:37:56.550092 27000 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:37:56.598080 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38393
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:56.928541 27013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:56.928807 27013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:56.928901 27013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:56.936352 27013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:56.936465 27013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:56.936578 27013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:37:56.946476 27013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:56.947726 27013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:56.949041 27013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:56.961835 27019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:56.962037 27018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:56.962170 27021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:56.963013 27013 server_base.cc:1034] running on GCE node
I20260806 20:37:56.963537 27013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:56.964443 27013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:56.965664 27013 hybrid_clock.cc:648] HybridClock initialized: now 1786048676965595 us; error 80 us; skew 500 ppm
I20260806 20:37:56.965945 27013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:56 dist-test-slave-tm1g krb5kdc[26905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048676, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:56.969044 27013 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:37:56.970469 27013 webserver.cc:460] Webserver started at http://127.23.200.1:42211/ using document root <none> and password file <none>
I20260806 20:37:56.971169 27013 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:56.971298 27013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:56.971626 27013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:56.974472 27013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "9c5fb82f0379402ea151e08a8e16a8bb"
format_stamp: "Formatted at 2026-08-06 20:37:56 on dist-test-slave-tm1g"
server_key: "e91d44f260bb79e7ba37726a2b6f1994"
server_key_iv: "de3c8979fc1da0abcce8b5835c34392f"
server_key_version: "encryptionkey@0"
I20260806 20:37:56.975284 27013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "9c5fb82f0379402ea151e08a8e16a8bb"
format_stamp: "Formatted at 2026-08-06 20:37:56 on dist-test-slave-tm1g"
server_key: "e91d44f260bb79e7ba37726a2b6f1994"
server_key_iv: "de3c8979fc1da0abcce8b5835c34392f"
server_key_version: "encryptionkey@0"
I20260806 20:37:56.980290 27013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:37:56.983687 27028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:56.984890 27013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:37:56.985049 27013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "9c5fb82f0379402ea151e08a8e16a8bb"
format_stamp: "Formatted at 2026-08-06 20:37:56 on dist-test-slave-tm1g"
server_key: "e91d44f260bb79e7ba37726a2b6f1994"
server_key_iv: "de3c8979fc1da0abcce8b5835c34392f"
server_key_version: "encryptionkey@0"
I20260806 20:37:56.985239 27013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:57.005684 27013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:57.009603 27013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:57.009884 27013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:57.010850 27013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:57.012121 27013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:57.012228 27013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:57.012374 27013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:57.012447 27013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:57.045498 27013 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:33581
I20260806 20:37:57.045583 27145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:33581 every 8 connection(s)
I20260806 20:37:57.047019 27013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:37:57.056736 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27013
I20260806 20:37:57.056941 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:37:57.057796 24352 external_mini_cluster.cc:1428] Setting key c3376ed84a9153cd901d5840014533be
Aug 06 20:37:57 dist-test-slave-tm1g krb5kdc[26905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048676, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:57.069327 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:57.049846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:57465 (local address 127.23.200.62:38393)
0806 20:37:57.050283 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:57.050287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:57.051209 (+ 922us) server_negotiation.cc:407] Connection header received
0806 20:37:57.053115 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:57.053121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:57.053226 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:57.053347 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:57.054895 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.055728 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.058226 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.058753 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.058903 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:57.062489 (+ 3586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:57.062497 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:57.062505 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:57.062549 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:57.066139 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:57.067096 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:57.067100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:57.067103 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:57.067225 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:57.067704 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:57.067708 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:57.067709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:57.067964 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:57.068074 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:57.068909 (+ 835us) server_negotiation.cc:299] Negotiation successful
0806 20:37:57.069076 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":234,"threads_started":1}
I20260806 20:37:57.070868 27149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:57.050275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38393 (local address 127.23.200.1:57465)
0806 20:37:57.051054 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:57.051096 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:57.052815 (+ 1719us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:57.053583 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:57.053592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:57.054092 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:57.054720 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:57.054742 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.055950 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.055953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:57.058067 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:57.058083 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.059020 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.059028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:57.059295 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:57.060043 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:57.060071 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:57.062226 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:57.066288 (+ 4062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:57.066321 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:57.066341 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:57.066967 (+ 626us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:57.067336 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:57.067343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:57.067350 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:57.067589 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:57.068086 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:57.068097 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:57.068524 (+ 427us) client_negotiation.cc:769] Sending connection context
0806 20:37:57.068813 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 20:37:57.069362 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":268,"threads_started":1}
I20260806 20:37:57.072459 27146 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38393
I20260806 20:37:57.072769 27146 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:57.073462 27146 heartbeater.cc:511] Master 127.23.200.62:38393 requested a full tablet report, sending...
I20260806 20:37:57.075912 26950 ts_manager.cc:194] Registered new tserver with Master: 9c5fb82f0379402ea151e08a8e16a8bb (127.23.200.1:33581)
I20260806 20:37:57.078112 26950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:57465
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:37:57.103269 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38393
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:57.422957 27152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:57.423256 27152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:57.423434 27152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:57.430445 27152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:57.430584 27152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:57.430773 27152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:37:57.440061 27152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:57.441320 27152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:57.442672 27152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:57.455641 27160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:57.455703 27158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:57.455677 27157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:57.456480 27152 server_base.cc:1034] running on GCE node
I20260806 20:37:57.457077 27152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:57.458042 27152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:57.459282 27152 hybrid_clock.cc:648] HybridClock initialized: now 1786048677459226 us; error 43 us; skew 500 ppm
I20260806 20:37:57.459635 27152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:57 dist-test-slave-tm1g krb5kdc[26905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048677, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:57.466032 27152 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:37:57.467561 27152 webserver.cc:460] Webserver started at http://127.23.200.2:42609/ using document root <none> and password file <none>
I20260806 20:37:57.468230 27152 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:57.468380 27152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:57.468683 27152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:57.471583 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "24db4e987ef74a47b4cf83ec1ac04f68"
format_stamp: "Formatted at 2026-08-06 20:37:57 on dist-test-slave-tm1g"
server_key: "155e330de0ad55aa831c1062ea144a6f"
server_key_iv: "78a213d883a4cb88f4c5daa7e92ac90e"
server_key_version: "encryptionkey@0"
I20260806 20:37:57.472352 27152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "24db4e987ef74a47b4cf83ec1ac04f68"
format_stamp: "Formatted at 2026-08-06 20:37:57 on dist-test-slave-tm1g"
server_key: "155e330de0ad55aa831c1062ea144a6f"
server_key_iv: "78a213d883a4cb88f4c5daa7e92ac90e"
server_key_version: "encryptionkey@0"
I20260806 20:37:57.477243 27152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:37:57.480634 27169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:57.482023 27152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:37:57.482205 27152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "24db4e987ef74a47b4cf83ec1ac04f68"
format_stamp: "Formatted at 2026-08-06 20:37:57 on dist-test-slave-tm1g"
server_key: "155e330de0ad55aa831c1062ea144a6f"
server_key_iv: "78a213d883a4cb88f4c5daa7e92ac90e"
server_key_version: "encryptionkey@0"
I20260806 20:37:57.482486 27152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:57.497377 27152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:57.501212 27152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:57.501502 27152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:57.502475 27152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:57.503813 27152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:57.503916 27152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:57.504060 27152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:57.504132 27152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:57.537365 27152 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:40423
I20260806 20:37:57.537413 27287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:40423 every 8 connection(s)
I20260806 20:37:57.538775 27152 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:37:57.539669 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27152
I20260806 20:37:57.539846 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:37:57.540414 24352 external_mini_cluster.cc:1428] Setting key 3f741927ca877f80a9363a48c03e6045
Aug 06 20:37:57 dist-test-slave-tm1g krb5kdc[26905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048677, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
I20260806 20:37:57.560808 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:57.541646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:49129 (local address 127.23.200.62:38393)
0806 20:37:57.541814 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:57.541819 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:57.543237 (+ 1418us) server_negotiation.cc:407] Connection header received
0806 20:37:57.545027 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:57.545032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:57.545130 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:57.545250 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:57.546795 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.547531 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.549490 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.549907 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.550018 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:57.554102 (+ 4084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:57.554107 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:57.554111 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:57.554143 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:57.557537 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:57.558261 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:57.558267 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:57.558271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:57.558457 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:57.558896 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:57.558907 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:57.558912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:57.559309 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:57.559476 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:57.560171 (+ 695us) server_negotiation.cc:299] Negotiation successful
0806 20:37:57.560441 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 20:37:57.561580 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:57.542242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38393 (local address 127.23.200.2:49129)
0806 20:37:57.543091 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:57.543134 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:57.544770 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:57.545498 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:57.545506 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:57.546012 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:57.546631 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:57.546648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.547695 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.547699 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:57.549309 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:57.549327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:57.550139 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:57.550148 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:57.550495 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:57.551208 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:57.551249 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:57.553869 (+ 2620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:57.557684 (+ 3815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:57.557690 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:57.557701 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:57.558120 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:57.558582 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:57.558586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:57.558588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:57.558758 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:57.559446 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:57.559452 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:57.559765 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 20:37:57.560024 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 20:37:57.560435 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
I20260806 20:37:57.563205 27288 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38393
I20260806 20:37:57.563510 27288 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:57.564220 27288 heartbeater.cc:511] Master 127.23.200.62:38393 requested a full tablet report, sending...
I20260806 20:37:57.565744 26950 ts_manager.cc:194] Registered new tserver with Master: 24db4e987ef74a47b4cf83ec1ac04f68 (127.23.200.2:40423)
I20260806 20:37:57.567013 26950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:49129
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:37:57.586440 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38393
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:57.913707 27293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:57.914017 27293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:57.914193 27293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:57.921273 27293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:57.921418 27293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:57.921649 27293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:37:57.931221 27293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38393
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:57.932554 27293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:57.933861 27293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:57.946887 27301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:57.946916 27298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:57.946952 27299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:57.947671 27293 server_base.cc:1034] running on GCE node
I20260806 20:37:57.948184 27293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:57.949012 27293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:57.950253 27293 hybrid_clock.cc:648] HybridClock initialized: now 1786048677950208 us; error 53 us; skew 500 ppm
I20260806 20:37:57.950631 27293 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:57 dist-test-slave-tm1g krb5kdc[26905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048677, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:57.953828 27293 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:37:57.955269 27293 webserver.cc:460] Webserver started at http://127.23.200.3:46553/ using document root <none> and password file <none>
I20260806 20:37:57.955921 27293 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:57.956020 27293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:57.956274 27293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:57.959468 27293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "758acb1243554693b3238a203bb686d4"
format_stamp: "Formatted at 2026-08-06 20:37:57 on dist-test-slave-tm1g"
server_key: "5421db1f6a2646ce599f83a8d11a981a"
server_key_iv: "7b32599aa02a907a8078c17ab9442d6f"
server_key_version: "encryptionkey@0"
I20260806 20:37:57.960199 27293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "758acb1243554693b3238a203bb686d4"
format_stamp: "Formatted at 2026-08-06 20:37:57 on dist-test-slave-tm1g"
server_key: "5421db1f6a2646ce599f83a8d11a981a"
server_key_iv: "7b32599aa02a907a8078c17ab9442d6f"
server_key_version: "encryptionkey@0"
I20260806 20:37:57.965215 27293 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 20:37:57.968725 27308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:57.970012 27293 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:37:57.970182 27293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "758acb1243554693b3238a203bb686d4"
format_stamp: "Formatted at 2026-08-06 20:37:57 on dist-test-slave-tm1g"
server_key: "5421db1f6a2646ce599f83a8d11a981a"
server_key_iv: "7b32599aa02a907a8078c17ab9442d6f"
server_key_version: "encryptionkey@0"
I20260806 20:37:57.970449 27293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:58.007884 27293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:58.011999 27293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:58.012298 27293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:37:58.013289 27293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:37:58.014683 27293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:37:58.014796 27293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:58.014940 27293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:37:58.015014 27293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:58.049022 27293 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:33627
I20260806 20:37:58.049114 27424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:33627 every 8 connection(s)
I20260806 20:37:58.050287 27293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:37:58.054064 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27293
I20260806 20:37:58.054234 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:37:58.054837 24352 external_mini_cluster.cc:1428] Setting key 7e0bf135400c6ce473b5a982fb30b230
Aug 06 20:37:58 dist-test-slave-tm1g krb5kdc[26905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048677, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:37:58.071020 27148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:58.052939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:44611 (local address 127.23.200.62:38393)
0806 20:37:58.053164 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:58.053170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:58.054391 (+ 1221us) server_negotiation.cc:407] Connection header received
0806 20:37:58.056411 (+ 2020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:58.056417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:58.056544 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:58.056656 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:58.058664 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:58.059416 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:58.061033 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:37:58.061428 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:37:58.061522 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:58.065030 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:37:58.065037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:37:58.065042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:37:58.065083 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:37:58.067634 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:58.068411 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:58.068415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:58.068418 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:58.068609 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:37:58.069154 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:37:58.069158 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:37:58.069160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:37:58.069510 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:37:58.069694 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 20:37:58.070582 (+ 888us) server_negotiation.cc:299] Negotiation successful
0806 20:37:58.070749 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:37:58.072049 27427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:37:58.053399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38393 (local address 127.23.200.3:44611)
0806 20:37:58.054192 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0806 20:37:58.054250 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 20:37:58.056151 (+ 1901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:37:58.056913 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:37:58.056921 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:37:58.057674 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:37:58.058506 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:58.058530 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:58.059584 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:58.059588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:58.060885 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:37:58.060897 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:37:58.061631 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:37:58.061637 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:37:58.061842 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:37:58.062601 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:37:58.062625 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:37:58.064817 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:37:58.067783 (+ 2966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:58.067791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:58.067807 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:58.068266 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:58.068759 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:37:58.068763 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:37:58.068766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:37:58.069020 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:37:58.069673 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:37:58.069700 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:37:58.070123 (+ 423us) client_negotiation.cc:769] Sending connection context
0806 20:37:58.070488 (+ 365us) client_negotiation.cc:240] Negotiation successful
0806 20:37:58.070929 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":348,"threads_started":1}
I20260806 20:37:58.073414 27425 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38393
I20260806 20:37:58.073767 27425 heartbeater.cc:464] Registering TS with master...
I20260806 20:37:58.074587 27425 heartbeater.cc:511] Master 127.23.200.62:38393 requested a full tablet report, sending...
I20260806 20:37:58.076033 26950 ts_manager.cc:194] Registered new tserver with Master: 758acb1243554693b3238a203bb686d4 (127.23.200.3:33627)
I20260806 20:37:58.076893 26950 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:44611
I20260806 20:37:58.081061 27146 heartbeater.cc:503] Master 127.23.200.62:38393 was elected leader, sending a full tablet report...
I20260806 20:37:58.082839 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 20:37:58.115664 27148 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:37:58.113074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.23.200.62:38393)
0806 20:37:58.113315 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 20:37:58.113323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:37:58.113694 (+ 371us) server_negotiation.cc:407] Connection header received
0806 20:37:58.113920 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:37:58.113926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:37:58.113994 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:37:58.114102 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:37:58.115452 (+ 1350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53338: BlockingRecv error: recv got EOF from 127.0.0.1:53338 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:37:58.119189 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27013
W20260806 20:37:58.122126 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27013: Not authorized: Client connection negotiation failed: client connection to 127.23.200.1:33581: server requires authentication, but client does not have Kerberos credentials available
I20260806 20:37:58.122304 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27013
I20260806 20:37:58.140810 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27152
W20260806 20:37:58.143527 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27152: Not authorized: Client connection negotiation failed: client connection to 127.23.200.2:40423: server requires authentication, but client does not have Kerberos credentials available
I20260806 20:37:58.143644 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27152
I20260806 20:37:58.160972 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27293
W20260806 20:37:58.164034 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27293: Not authorized: Client connection negotiation failed: client connection to 127.23.200.3:33627: server requires authentication, but client does not have Kerberos credentials available
I20260806 20:37:58.164212 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27293
I20260806 20:37:58.180606 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 26918
I20260806 20:37:58.181834 26981 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:37:58.286899 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 26918
2026-08-06T20:37:58Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3553 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:37:58 dist-test-slave-tm1g krb5kdc[27442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:37:58 dist-test-slave-tm1g krb5kdc[27442](info): set up 2 sockets
Aug 06 20:37:58 dist-test-slave-tm1g krb5kdc[27442](info): commencing operation
krb5kdc: starting...
W20260806 20:37:59.393690 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.000s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:37:59 dist-test-slave-tm1g krb5kdc[27442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:37:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:37:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:37:59.596068 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36703
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:34095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:36703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:37:59.913628 27456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:37:59.913892 27456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:37:59.914050 27456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:37:59.921021 27456 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:37:59.921137 27456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:37:59.921240 27456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:37:59.921342 27456 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:37:59.921427 27456 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:37:59.930827 27456 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:34095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:36703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:37:59.932019 27456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:37:59.933265 27456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:37:59.945062 27464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:59.945098 27462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:37:59.945104 27461 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:37:59.945945 27456 server_base.cc:1034] running on GCE node
I20260806 20:37:59.946772 27456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:37:59.948053 27456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:37:59.949287 27456 hybrid_clock.cc:648] HybridClock initialized: now 1786048679949249 us; error 47 us; skew 500 ppm
I20260806 20:37:59.949642 27456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:37:59 dist-test-slave-tm1g krb5kdc[27442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048679, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:37:59.952692 27456 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:37:59.953999 27456 webserver.cc:460] Webserver started at http://127.23.200.62:35707/ using document root <none> and password file <none>
I20260806 20:37:59.954695 27456 fs_manager.cc:359] Metadata directory not provided
I20260806 20:37:59.954802 27456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:37:59.955058 27456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:37:59.957904 27456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "425d57403e6b46b696b07374484bb829"
format_stamp: "Formatted at 2026-08-06 20:37:59 on dist-test-slave-tm1g"
server_key: "b2d6115bc8511fb26dc33f133f1ea281"
server_key_iv: "d61776db8c78fda17128f5fe4ec93e9d"
server_key_version: "encryptionkey@0"
I20260806 20:37:59.958693 27456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "425d57403e6b46b696b07374484bb829"
format_stamp: "Formatted at 2026-08-06 20:37:59 on dist-test-slave-tm1g"
server_key: "b2d6115bc8511fb26dc33f133f1ea281"
server_key_iv: "d61776db8c78fda17128f5fe4ec93e9d"
server_key_version: "encryptionkey@0"
I20260806 20:37:59.963734 27456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 20:37:59.966949 27471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:37:59.968179 27456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:37:59.968309 27456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "425d57403e6b46b696b07374484bb829"
format_stamp: "Formatted at 2026-08-06 20:37:59 on dist-test-slave-tm1g"
server_key: "b2d6115bc8511fb26dc33f133f1ea281"
server_key_iv: "d61776db8c78fda17128f5fe4ec93e9d"
server_key_version: "encryptionkey@0"
I20260806 20:37:59.968487 27456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:37:59.991158 27456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:37:59.994879 27456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:37:59.995105 27456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:00.013461 27456 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:36703
I20260806 20:38:00.013504 27524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:36703 every 8 connection(s)
I20260806 20:38:00.014806 27456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:00.018507 27525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:00.022354 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27456
I20260806 20:38:00.022629 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:00.023430 24352 external_mini_cluster.cc:1428] Setting key 98fc3b71e27b359847e91539153488ab
I20260806 20:38:00.031930 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829: Bootstrap starting.
I20260806 20:38:00.035712 27525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:00.037072 27525 log.cc:824] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829: Log is configured to *not* fsync() on all Append() calls
Aug 06 20:38:00 dist-test-slave-tm1g krb5kdc[27442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:00.040177 27525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829: No bootstrap required, opened a new log
I20260806 20:38:00.047788 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:00.025401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.23.200.62:36703)
0806 20:38:00.026161 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:00.026172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:00.026211 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:38:00.027827 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:00.027848 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:00.028229 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:00.028543 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:00.030538 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:00.031812 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:00.033576 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:00.034234 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:00.034437 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:00.039297 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:00.039308 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:00.039327 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:00.039379 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:00.043478 (+ 4099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:00.044151 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:00.044159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:00.044166 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:00.044385 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:00.044735 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:00.044740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:00.044745 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:00.045435 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:00.045758 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:00.046043 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 20:38:00.046493 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":359,"threads_started":1}
I20260806 20:38:00.050376 27525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425d57403e6b46b696b07374484bb829" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36703 } }
I20260806 20:38:00.050806 27525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:00.050952 27525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425d57403e6b46b696b07374484bb829, State: Initialized, Role: FOLLOWER
I20260806 20:38:00.051595 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:38:00.051600 27525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425d57403e6b46b696b07374484bb829" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36703 } }
I20260806 20:38:00.051888 27525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:00.052042 27525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:00.052228 27525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:00.054291 27525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425d57403e6b46b696b07374484bb829" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36703 } }
I20260806 20:38:00.054826 27525 leader_election.cc:302] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 425d57403e6b46b696b07374484bb829; no voters:
I20260806 20:38:00.055403 27525 leader_election.cc:288] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:00.055589 27530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:00.056177 27530 raft_consensus.cc:686] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [term 1 LEADER]: Becoming Leader. State: Replica: 425d57403e6b46b696b07374484bb829, State: Running, Role: LEADER
I20260806 20:38:00.056814 27530 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425d57403e6b46b696b07374484bb829" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36703 } }
I20260806 20:38:00.059437 27532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425d57403e6b46b696b07374484bb829. Latest consensus state: current_term: 1 leader_uuid: "425d57403e6b46b696b07374484bb829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425d57403e6b46b696b07374484bb829" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36703 } } }
I20260806 20:38:00.059813 27532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:00.067015 27530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "425d57403e6b46b696b07374484bb829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "425d57403e6b46b696b07374484bb829" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36703 } } }
I20260806 20:38:00.067762 27530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:00.072364 27525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:00.072685 27535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:00.079025 27535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:00.093024 27535 catalog_manager.cc:1347] Generated new cluster ID: d35b9e6b8fb74f529e87acf7f6db9f33
I20260806 20:38:00.093146 27535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:00.121169 27535 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:00.123413 27535 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:00.133237 27528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:00.123430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58078 (local address 127.23.200.62:36703)
0806 20:38:00.123614 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:00.123620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:00.124214 (+ 594us) server_negotiation.cc:407] Connection header received
0806 20:38:00.124460 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:00.124465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:00.124563 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:00.124677 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:00.125959 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:00.127163 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:00.128996 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:00.129655 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:00.129752 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:00.131412 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:00.131419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:00.131423 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:00.132276 (+ 853us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:00.132370 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:00.132407 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:00.132502 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:00.132775 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 20:38:00.132873 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:38:00.134130 27535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 425d57403e6b46b696b07374484bb829: Generated new TSK 0
I20260806 20:38:00.135241 27535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:38:00.147137 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27456
I20260806 20:38:00.148392 27520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:00.236883 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27456
2026-08-06T20:38:00Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1943 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:00 dist-test-slave-tm1g krb5kdc[27562](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:00 dist-test-slave-tm1g krb5kdc[27562](info): set up 2 sockets
Aug 06 20:38:00 dist-test-slave-tm1g krb5kdc[27562](info): commencing operation
krb5kdc: starting...
W20260806 20:38:01.334357 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.004s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:01 dist-test-slave-tm1g krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:38:01.530350 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:39237
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:39237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:01.850394 27575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:01.850667 27575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:01.850878 27575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:01.857895 27575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:01.857990 27575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:01.858100 27575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:01.858186 27575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:01.858263 27575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:01.868112 27575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:39237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:39237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:01.869413 27575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:01.870749 27575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:01.882892 27583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:01.883042 27581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:01.883011 27580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:01.883890 27575 server_base.cc:1034] running on GCE node
I20260806 20:38:01.884575 27575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:01.885872 27575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:01.887071 27575 hybrid_clock.cc:648] HybridClock initialized: now 1786048681887040 us; error 31 us; skew 500 ppm
I20260806 20:38:01.887377 27575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:01 dist-test-slave-tm1g krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048681, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:01.890290 27575 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:38:01.891674 27575 webserver.cc:460] Webserver started at http://127.23.200.62:44941/ using document root <none> and password file <none>
I20260806 20:38:01.892347 27575 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:01.892452 27575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:01.892702 27575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:01.895537 27575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "7ba3544797c14ac982a0b198bdfcd8a6"
format_stamp: "Formatted at 2026-08-06 20:38:01 on dist-test-slave-tm1g"
server_key: "db176443c44d25b947d1ea6c264dc166"
server_key_iv: "bb5b55ad4e4162541c24f755eb1e2067"
server_key_version: "encryptionkey@0"
I20260806 20:38:01.896229 27575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "7ba3544797c14ac982a0b198bdfcd8a6"
format_stamp: "Formatted at 2026-08-06 20:38:01 on dist-test-slave-tm1g"
server_key: "db176443c44d25b947d1ea6c264dc166"
server_key_iv: "bb5b55ad4e4162541c24f755eb1e2067"
server_key_version: "encryptionkey@0"
I20260806 20:38:01.901186 27575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 20:38:01.904407 27590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:01.905629 27575 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 20:38:01.905762 27575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "7ba3544797c14ac982a0b198bdfcd8a6"
format_stamp: "Formatted at 2026-08-06 20:38:01 on dist-test-slave-tm1g"
server_key: "db176443c44d25b947d1ea6c264dc166"
server_key_iv: "bb5b55ad4e4162541c24f755eb1e2067"
server_key_version: "encryptionkey@0"
I20260806 20:38:01.905910 27575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:01.932406 27575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:01.936357 27575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:01.936623 27575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:01.954507 27575 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:39237
I20260806 20:38:01.954535 27645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:39237 every 8 connection(s)
I20260806 20:38:01.955782 27575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:01.959429 27646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:01.963629 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27575
I20260806 20:38:01.963862 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:01.964493 24352 external_mini_cluster.cc:1428] Setting key f13d4e69ee670f936dfbc0460c67eb4c
I20260806 20:38:01.970188 27646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6: Bootstrap starting.
I20260806 20:38:01.973232 27646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:01.974280 27646 log.cc:824] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:01.977140 27646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6: No bootstrap required, opened a new log
Aug 06 20:38:01 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:01.985255 27646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39237 } }
I20260806 20:38:01.985658 27646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:01.985782 27646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ba3544797c14ac982a0b198bdfcd8a6, State: Initialized, Role: FOLLOWER
I20260806 20:38:01.986143 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:01.966483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.23.200.62:39237)
0806 20:38:01.967290 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:01.967301 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:01.967343 (+ 42us) server_negotiation.cc:407] Connection header received
0806 20:38:01.969018 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:01.969040 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:01.969413 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:01.969773 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:01.971227 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:01.972611 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:01.973894 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:01.974672 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:01.974875 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:01.978868 (+ 3993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:01.978876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:01.978893 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:01.978926 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:01.982040 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:01.982826 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:01.982832 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:01.982837 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:01.983016 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:01.983427 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:01.983430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:01.983432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:01.983928 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:01.984285 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:01.984643 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 20:38:01.985044 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":361,"threads_started":1}
I20260806 20:38:01.986464 27646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39237 } }
I20260806 20:38:01.986717 27646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:01.986893 27646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:01.987098 27646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:01.989496 27646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39237 } }
I20260806 20:38:01.989969 27646 leader_election.cc:302] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ba3544797c14ac982a0b198bdfcd8a6; no voters:
I20260806 20:38:01.990561 27646 leader_election.cc:288] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:01.990688 27651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:01.991357 27651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [term 1 LEADER]: Becoming Leader. State: Replica: 7ba3544797c14ac982a0b198bdfcd8a6, State: Running, Role: LEADER
I20260806 20:38:01.991860 27651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39237 } }
I20260806 20:38:01.992583 27646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:01.994560 27653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ba3544797c14ac982a0b198bdfcd8a6. Latest consensus state: current_term: 1 leader_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39237 } } }
I20260806 20:38:01.994720 27652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ba3544797c14ac982a0b198bdfcd8a6" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39237 } } }
I20260806 20:38:01.995417 27653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:01.995468 27652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:01.996460 27659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:02.002501 27659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:02.015125 27659 catalog_manager.cc:1347] Generated new cluster ID: 21dcc7df4d5b4e91a071b73a582a6838
I20260806 20:38:02.015237 27659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:02.044445 27659 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:02.046253 27659 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:02.056051 27659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ba3544797c14ac982a0b198bdfcd8a6: Generated new TSK 0
I20260806 20:38:02.057014 27659 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:38:02.106703 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39237
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:02.429307 27672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:02.429618 27672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:02.429778 27672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:02.436748 27672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:02.436887 27672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:02.437067 27672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:02.446339 27672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:02.447576 27672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:02.448851 27672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:02.461659 27677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:02.461715 27678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:02.461616 27680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:02.462637 27672 server_base.cc:1034] running on GCE node
I20260806 20:38:02.463239 27672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:02.464128 27672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:02.465330 27672 hybrid_clock.cc:648] HybridClock initialized: now 1786048682465281 us; error 31 us; skew 500 ppm
I20260806 20:38:02.465631 27672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:02 dist-test-slave-tm1g krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:02.472915 27672 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:38:02.474406 27672 webserver.cc:460] Webserver started at http://127.23.200.1:38371/ using document root <none> and password file <none>
I20260806 20:38:02.475075 27672 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:02.475220 27672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:02.475529 27672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:02.478382 27672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "5ceeb395e792418f8b84cb476705c314"
format_stamp: "Formatted at 2026-08-06 20:38:02 on dist-test-slave-tm1g"
server_key: "2d360ff4aff48c87819f228e95417d1d"
server_key_iv: "056ec4b1b573742f84fdd611d20966ee"
server_key_version: "encryptionkey@0"
I20260806 20:38:02.479162 27672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "5ceeb395e792418f8b84cb476705c314"
format_stamp: "Formatted at 2026-08-06 20:38:02 on dist-test-slave-tm1g"
server_key: "2d360ff4aff48c87819f228e95417d1d"
server_key_iv: "056ec4b1b573742f84fdd611d20966ee"
server_key_version: "encryptionkey@0"
I20260806 20:38:02.484081 27672 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:38:02.487730 27687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:02.488948 27672 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:38:02.489125 27672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "5ceeb395e792418f8b84cb476705c314"
format_stamp: "Formatted at 2026-08-06 20:38:02 on dist-test-slave-tm1g"
server_key: "2d360ff4aff48c87819f228e95417d1d"
server_key_iv: "056ec4b1b573742f84fdd611d20966ee"
server_key_version: "encryptionkey@0"
I20260806 20:38:02.489329 27672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:02.514767 27672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:02.518461 27672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:02.518754 27672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:02.519701 27672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:02.520943 27672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:02.521054 27672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:02.521209 27672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:02.521281 27672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:02.554019 27672 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:41759
I20260806 20:38:02.554100 27806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:41759 every 8 connection(s)
I20260806 20:38:02.555303 27672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:02.555783 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27672
I20260806 20:38:02.555996 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:02.556851 24352 external_mini_cluster.cc:1428] Setting key 071c25de85dea6adabb508a4bf6b5737
Aug 06 20:38:02 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:38:02.578610 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:02.557906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:45131 (local address 127.23.200.62:39237)
0806 20:38:02.558416 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:02.558423 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:02.559638 (+ 1215us) server_negotiation.cc:407] Connection header received
0806 20:38:02.561434 (+ 1796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:02.561442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:02.561549 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:02.561668 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:02.563159 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:02.564314 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:02.566814 (+ 2500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:02.567378 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:02.567476 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:02.571639 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:02.571647 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:02.571655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:02.571698 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:02.575410 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:02.576117 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:02.576123 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:02.576128 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:02.576288 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:02.576701 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:02.576707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:02.576710 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:02.577102 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:02.577266 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:02.577913 (+ 647us) server_negotiation.cc:299] Negotiation successful
0806 20:38:02.578148 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":273,"threads_started":1}
I20260806 20:38:02.579267 27810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:02.558584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:39237 (local address 127.23.200.1:45131)
0806 20:38:02.559477 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:02.559522 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:02.561156 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:02.561853 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:02.561862 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:02.562439 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:02.562981 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:02.562998 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:02.564466 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:02.564470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:02.566665 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:02.566682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:02.567628 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:02.567659 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:02.567984 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:02.568721 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:02.568753 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:02.571410 (+ 2657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:02.575550 (+ 4140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:02.575557 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:02.575567 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:02.575980 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:02.576398 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:02.576401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:02.576403 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:02.576578 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:02.577247 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:02.577252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:02.577529 (+ 277us) client_negotiation.cc:769] Sending connection context
0806 20:38:02.577732 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 20:38:02.578071 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":353,"threads_started":1}
I20260806 20:38:02.580734 27807 heartbeater.cc:347] Connected to a master server at 127.23.200.62:39237
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
I20260806 20:38:02.581175 27807 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:02.582079 27807 heartbeater.cc:511] Master 127.23.200.62:39237 requested a full tablet report, sending...
I20260806 20:38:02.584326 27607 ts_manager.cc:194] Registered new tserver with Master: 5ceeb395e792418f8b84cb476705c314 (127.23.200.1:41759)
I20260806 20:38:02.585789 27607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:45131
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:38:02.605234 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39237
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:02.928506 27813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:02.928793 27813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:02.928969 27813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:02.936249 27813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:02.936388 27813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:02.936589 27813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:02.946405 27813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:02.947914 27813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:02.949307 27813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:02.963245 27819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:02.963337 27820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:02.963960 27822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:02.964237 27813 server_base.cc:1034] running on GCE node
I20260806 20:38:02.964884 27813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:02.965943 27813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:02.967171 27813 hybrid_clock.cc:648] HybridClock initialized: now 1786048682967102 us; error 60 us; skew 500 ppm
I20260806 20:38:02.967537 27813 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:02 dist-test-slave-tm1g krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:02.971263 27813 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:38:02.975026 27813 webserver.cc:460] Webserver started at http://127.23.200.2:34325/ using document root <none> and password file <none>
I20260806 20:38:02.975721 27813 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:02.975867 27813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:02.976176 27813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:02.979081 27813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "ec64b39d1d8d48da9199c7e803d821ce"
format_stamp: "Formatted at 2026-08-06 20:38:02 on dist-test-slave-tm1g"
server_key: "47ba86685f42d295a72549dca571915f"
server_key_iv: "f983a19f886f422e243dac444d8fbaa0"
server_key_version: "encryptionkey@0"
I20260806 20:38:02.979866 27813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "ec64b39d1d8d48da9199c7e803d821ce"
format_stamp: "Formatted at 2026-08-06 20:38:02 on dist-test-slave-tm1g"
server_key: "47ba86685f42d295a72549dca571915f"
server_key_iv: "f983a19f886f422e243dac444d8fbaa0"
server_key_version: "encryptionkey@0"
I20260806 20:38:02.984905 27813 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:38:02.988278 27829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:02.989593 27813 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:38:02.989784 27813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "ec64b39d1d8d48da9199c7e803d821ce"
format_stamp: "Formatted at 2026-08-06 20:38:02 on dist-test-slave-tm1g"
server_key: "47ba86685f42d295a72549dca571915f"
server_key_iv: "f983a19f886f422e243dac444d8fbaa0"
server_key_version: "encryptionkey@0"
I20260806 20:38:02.990006 27813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:03.003330 27813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:03.007120 27813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:03.007421 27813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:03.008355 27813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:03.009814 27813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:03.009927 27813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:03.010092 27813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:03.010162 27813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:03.043879 27813 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:42635
I20260806 20:38:03.043915 27948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:42635 every 8 connection(s)
I20260806 20:38:03.045176 27813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:03.053164 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27813
I20260806 20:38:03.053372 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:03.053817 24352 external_mini_cluster.cc:1428] Setting key 6d90ac427568f8bf8d0f63f68f5bbb75
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048682, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:03.065977 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.047861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:46993 (local address 127.23.200.62:39237)
0806 20:38:03.048047 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:03.048052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:03.049193 (+ 1141us) server_negotiation.cc:407] Connection header received
0806 20:38:03.050969 (+ 1776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:03.050980 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:03.051053 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:03.051143 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:03.052576 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.053389 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.055158 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.055725 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.055860 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.059397 (+ 3537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:03.059402 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:03.059405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:03.059436 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:03.062765 (+ 3329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.063477 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.063486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.063492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.063656 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.064073 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.064080 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.064086 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.064449 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:03.064600 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:03.065213 (+ 613us) server_negotiation.cc:299] Negotiation successful
0806 20:38:03.065507 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 20:38:03.066560 27951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.048230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:39237 (local address 127.23.200.2:46993)
0806 20:38:03.048996 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:03.049045 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:03.050681 (+ 1636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:03.051339 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:03.051346 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:03.051831 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:03.052409 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:03.052426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.053555 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.053560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:03.055013 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:03.055026 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.055952 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.055959 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:03.056198 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.057018 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:03.057047 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:03.059177 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:03.062935 (+ 3758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:03.062941 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:03.062951 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:03.063343 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:03.063772 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:03.063775 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:03.063778 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:03.063957 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:03.064563 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:03.064568 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:03.064848 (+ 280us) client_negotiation.cc:769] Sending connection context
0806 20:38:03.065061 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 20:38:03.065445 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":311,"threads_started":1}
I20260806 20:38:03.068202 27949 heartbeater.cc:347] Connected to a master server at 127.23.200.62:39237
I20260806 20:38:03.068542 27949 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:03.069237 27949 heartbeater.cc:511] Master 127.23.200.62:39237 requested a full tablet report, sending...
I20260806 20:38:03.070848 27607 ts_manager.cc:194] Registered new tserver with Master: ec64b39d1d8d48da9199c7e803d821ce (127.23.200.2:42635)
I20260806 20:38:03.072181 27607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:46993
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:38:03.099722 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39237
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:03.420866 27954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:03.421139 27954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:03.421232 27954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:03.428220 27954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:03.428311 27954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:03.428444 27954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:03.437678 27954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:03.439073 27954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:03.440311 27954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:03.453428 27963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:03.453377 27959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:03.453488 27960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:03.454288 27954 server_base.cc:1034] running on GCE node
I20260806 20:38:03.454911 27954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:03.455813 27954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:03.457033 27954 hybrid_clock.cc:648] HybridClock initialized: now 1786048683456999 us; error 50 us; skew 500 ppm
I20260806 20:38:03.457326 27954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:03.460273 27954 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:38:03.461725 27954 webserver.cc:460] Webserver started at http://127.23.200.3:44419/ using document root <none> and password file <none>
I20260806 20:38:03.462441 27954 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:03.462589 27954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:03.462898 27954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:03.465749 27954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "7259a0d263574541987904d75d87ff0e"
format_stamp: "Formatted at 2026-08-06 20:38:03 on dist-test-slave-tm1g"
server_key: "b1d8c38b3c0cbb3611b9c1c949e41e77"
server_key_iv: "c73c91007dd40fe9a3ac4354d0a4f631"
server_key_version: "encryptionkey@0"
I20260806 20:38:03.466583 27954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "7259a0d263574541987904d75d87ff0e"
format_stamp: "Formatted at 2026-08-06 20:38:03 on dist-test-slave-tm1g"
server_key: "b1d8c38b3c0cbb3611b9c1c949e41e77"
server_key_iv: "c73c91007dd40fe9a3ac4354d0a4f631"
server_key_version: "encryptionkey@0"
I20260806 20:38:03.471601 27954 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 20:38:03.474889 27970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:03.476177 27954 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:03.476341 27954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "7259a0d263574541987904d75d87ff0e"
format_stamp: "Formatted at 2026-08-06 20:38:03 on dist-test-slave-tm1g"
server_key: "b1d8c38b3c0cbb3611b9c1c949e41e77"
server_key_iv: "c73c91007dd40fe9a3ac4354d0a4f631"
server_key_version: "encryptionkey@0"
I20260806 20:38:03.476533 27954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:03.495880 27954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:03.499930 27954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:03.500211 27954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:03.501108 27954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:03.502413 27954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:03.502518 27954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:03.502669 27954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:03.502743 27954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:03.535521 27954 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37199
I20260806 20:38:03.535568 28088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37199 every 8 connection(s)
I20260806 20:38:03.536818 27954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:03.546953 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 27954
I20260806 20:38:03.547174 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:03.547705 24352 external_mini_cluster.cc:1428] Setting key 9bf2e9a11626911c3b93ebe363ce345d
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:03.556380 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.539317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:53825 (local address 127.23.200.62:39237)
0806 20:38:03.539550 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:03.539556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:03.540650 (+ 1094us) server_negotiation.cc:407] Connection header received
0806 20:38:03.542400 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:03.542405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:03.542474 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:03.542565 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:03.544057 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.544803 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.546289 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.546754 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.546910 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.550678 (+ 3768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:03.550689 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:03.550692 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:03.550751 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:03.553440 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.554173 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.554180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.554185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.554373 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.554781 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.554786 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.554790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.555046 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:03.555190 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:03.555809 (+ 619us) server_negotiation.cc:299] Negotiation successful
0806 20:38:03.556017 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 20:38:03.557267 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.539790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:39237 (local address 127.23.200.3:53825)
0806 20:38:03.540495 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:03.540538 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:03.542097 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:03.542759 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:03.542766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:03.543267 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:03.543795 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:03.543810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.544937 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.544940 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:03.546160 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:03.546171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.546977 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.546988 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:03.547239 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.547993 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:03.548020 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:03.550487 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:03.553612 (+ 3125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:03.553620 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:03.553634 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:03.554044 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:03.554480 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:03.554485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:03.554488 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:03.554669 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:03.555176 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:03.555183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:03.555474 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 20:38:03.555723 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 20:38:03.556223 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":269,"threads_started":1}
I20260806 20:38:03.558846 28089 heartbeater.cc:347] Connected to a master server at 127.23.200.62:39237
I20260806 20:38:03.559124 28089 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:03.559808 28089 heartbeater.cc:511] Master 127.23.200.62:39237 requested a full tablet report, sending...
I20260806 20:38:03.561113 27607 ts_manager.cc:194] Registered new tserver with Master: 7259a0d263574541987904d75d87ff0e (127.23.200.3:37199)
I20260806 20:38:03.562127 27607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:53825
I20260806 20:38:03.563067 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:38:03.588538 27807 heartbeater.cc:503] Master 127.23.200.62:39237 was elected leader, sending a full tablet report...
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:03.610903 27809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.597432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.23.200.62:39237)
0806 20:38:03.597602 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:03.597607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:03.598174 (+ 567us) server_negotiation.cc:407] Connection header received
0806 20:38:03.598354 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:03.598359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:03.598422 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:03.598510 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:03.599818 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.600536 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.601697 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.602125 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.602219 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.605930 (+ 3711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:03.605935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:03.605938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:03.605970 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:03.608412 (+ 2442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.609075 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.609078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.609081 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.609205 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.609592 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.609595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.609597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.609967 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:03.610152 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:03.610462 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 20:38:03.610620 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260806 20:38:03.611452 27606 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40922
I20260806 20:38:03.613631 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27672
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:03.628715 28100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.614712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.23.200.1:41759)
0806 20:38:03.615402 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:03.615409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:03.615433 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:38:03.615486 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:03.615492 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:03.615689 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:03.615833 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:03.616891 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.617898 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.619115 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.619702 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.619801 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.623528 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:03.623539 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:03.623552 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:03.623589 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:03.626190 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.626946 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.626953 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.626955 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.627092 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.627439 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.627442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.627444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.627768 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:03.627995 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:03.628237 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 20:38:03.628437 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":360,"threads_started":1}
W20260806 20:38:03.629980 27802 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51736
W20260806 20:38:03.631079 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27672: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:38:03.631278 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27672
I20260806 20:38:03.649770 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27813
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:38:03.666985 28101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.650598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47154 (local address 127.23.200.2:42635)
0806 20:38:03.651216 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:03.651223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:03.651247 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:38:03.651305 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:03.651311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:03.651559 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:03.651689 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:03.653065 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.654114 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.655483 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.656039 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.656145 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.660192 (+ 4047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:03.660203 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:03.660216 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:03.660249 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:03.663446 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.664294 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.664302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.664305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.664483 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.664962 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.664970 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.664975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.665473 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:03.665811 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:03.666246 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 20:38:03.666677 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":334,"threads_started":1}
W20260806 20:38:03.668286 27944 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47154
W20260806 20:38:03.669334 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27813: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:38:03.669548 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27813
I20260806 20:38:03.689142 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27954
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[27562](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048683, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:38:03.705345 28102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:03.689999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52676 (local address 127.23.200.3:37199)
0806 20:38:03.690630 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:03.690637 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:03.690666 (+ 29us) server_negotiation.cc:407] Connection header received
0806 20:38:03.690723 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:03.690729 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:03.690934 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:03.691071 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:03.692374 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.693568 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.694888 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:03.695431 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:03.695532 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:03.700002 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:03.700012 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:03.700036 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:03.700070 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:03.702963 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.703535 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.703542 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.703545 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.703661 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:03.703980 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:03.703983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:03.703985 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:03.704301 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:03.704530 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:03.704734 (+ 204us) server_negotiation.cc:299] Negotiation successful
0806 20:38:03.704923 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":326,"threads_started":1}
W20260806 20:38:03.706650 28084 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52676
W20260806 20:38:03.707728 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27954: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:38:03.707890 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27954
I20260806 20:38:03.724820 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 27575
I20260806 20:38:03.726009 27639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:03.843514 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 27575
2026-08-06T20:38:03Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3613 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[28106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[28106](info): set up 2 sockets
Aug 06 20:38:03 dist-test-slave-tm1g krb5kdc[28106](info): commencing operation
krb5kdc: starting...
W20260806 20:38:04.961951 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:05 dist-test-slave-tm1g krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:38:05.161849 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:34243
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:34243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:05.493854 28119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:05.494186 28119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:05.494401 28119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:05.502444 28119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:05.502559 28119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:05.502667 28119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:05.502759 28119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:05.502825 28119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:05.513015 28119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:34243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:34243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:05.514513 28119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:05.515925 28119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:05.527966 28124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:05.528008 28128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:05.527968 28126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:05.528936 28119 server_base.cc:1034] running on GCE node
I20260806 20:38:05.529686 28119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:05.531205 28119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:05.532437 28119 hybrid_clock.cc:648] HybridClock initialized: now 1786048685532390 us; error 41 us; skew 500 ppm
I20260806 20:38:05.532764 28119 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:05 dist-test-slave-tm1g krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048685, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:05.535835 28119 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:38:05.537403 28119 webserver.cc:460] Webserver started at http://127.23.200.62:42749/ using document root <none> and password file <none>
I20260806 20:38:05.538136 28119 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:05.538283 28119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:05.538651 28119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:05.541797 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "695cd7a5bb5a4ffd922379d3152cef5c"
format_stamp: "Formatted at 2026-08-06 20:38:05 on dist-test-slave-tm1g"
server_key: "b99f43251bb3a1f4a0774840f389cba2"
server_key_iv: "4afd23639e283ccba5bcd3dc28483198"
server_key_version: "encryptionkey@0"
I20260806 20:38:05.542672 28119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "695cd7a5bb5a4ffd922379d3152cef5c"
format_stamp: "Formatted at 2026-08-06 20:38:05 on dist-test-slave-tm1g"
server_key: "b99f43251bb3a1f4a0774840f389cba2"
server_key_iv: "4afd23639e283ccba5bcd3dc28483198"
server_key_version: "encryptionkey@0"
I20260806 20:38:05.548121 28119 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:38:05.551789 28135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:05.553174 28119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:38:05.553376 28119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "695cd7a5bb5a4ffd922379d3152cef5c"
format_stamp: "Formatted at 2026-08-06 20:38:05 on dist-test-slave-tm1g"
server_key: "b99f43251bb3a1f4a0774840f389cba2"
server_key_iv: "4afd23639e283ccba5bcd3dc28483198"
server_key_version: "encryptionkey@0"
I20260806 20:38:05.553607 28119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:05.576546 28119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:05.580853 28119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:05.581168 28119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:05.600641 28119 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:34243
I20260806 20:38:05.600666 28189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:34243 every 8 connection(s)
I20260806 20:38:05.602159 28119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:05.603032 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28119
I20260806 20:38:05.603293 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:05.604007 24352 external_mini_cluster.cc:1428] Setting key 93b5690f31998bde8a5d626ad9a3e188
I20260806 20:38:05.607225 28190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 20:38:05 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048685, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:05.621559 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c: Bootstrap starting.
I20260806 20:38:05.626190 28190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:05.627627 28190 log.cc:824] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:05.630795 28193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:05.606034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45364 (local address 127.23.200.62:34243)
0806 20:38:05.607000 (+ 966us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:05.607016 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:05.607068 (+ 52us) server_negotiation.cc:407] Connection header received
0806 20:38:05.609556 (+ 2488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:05.609588 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:05.610098 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:05.610613 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:05.612038 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:05.614190 (+ 2152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:05.615606 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:05.616675 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:05.616859 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:05.621617 (+ 4758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:05.621630 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:05.621653 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:05.621709 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:05.625827 (+ 4118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:05.626719 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:05.626734 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:05.626744 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:05.626958 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:05.627332 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:05.627339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:05.627343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:05.628119 (+ 776us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:05.628511 (+ 392us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:05.628934 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 20:38:05.629384 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":504,"threads_started":1}
I20260806 20:38:05.631439 28190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c: No bootstrap required, opened a new log
I20260806 20:38:05.639571 28190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34243 } }
I20260806 20:38:05.639950 28190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:05.640028 28190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 695cd7a5bb5a4ffd922379d3152cef5c, State: Initialized, Role: FOLLOWER
I20260806 20:38:05.640779 28190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34243 } }
I20260806 20:38:05.641013 28190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:05.641132 28190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:05.641263 28190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:05.643537 28190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34243 } }
I20260806 20:38:05.644204 28190 leader_election.cc:302] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 695cd7a5bb5a4ffd922379d3152cef5c; no voters:
I20260806 20:38:05.644950 28190 leader_election.cc:288] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:05.645123 28195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:05.645882 28195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [term 1 LEADER]: Becoming Leader. State: Replica: 695cd7a5bb5a4ffd922379d3152cef5c, State: Running, Role: LEADER
I20260806 20:38:05.646598 28195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34243 } }
I20260806 20:38:05.647264 28190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:05.649372 28196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34243 } } }
I20260806 20:38:05.649919 28196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:05.649618 28198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 695cd7a5bb5a4ffd922379d3152cef5c. Latest consensus state: current_term: 1 leader_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "695cd7a5bb5a4ffd922379d3152cef5c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34243 } } }
I20260806 20:38:05.650791 28198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:05.651733 28201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:05.657128 28201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:05.667265 28201 catalog_manager.cc:1347] Generated new cluster ID: b1b7ad3949d54b2ab90f00176f39eaa6
I20260806 20:38:05.667407 28201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:05.677054 28201 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:05.678390 28201 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:05.688195 28201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 695cd7a5bb5a4ffd922379d3152cef5c: Generated new TSK 0
I20260806 20:38:05.689029 28201 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:38:05.737601 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34243
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:06.068992 28217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:06.069267 28217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:06.069458 28217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:06.077162 28217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:06.077284 28217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:06.077461 28217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:06.087459 28217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:06.088732 28217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:06.090023 28217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:06.102828 28223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:06.102929 28224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:06.103360 28226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:06.103832 28217 server_base.cc:1034] running on GCE node
I20260806 20:38:06.104375 28217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:06.105263 28217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:06.106495 28217 hybrid_clock.cc:648] HybridClock initialized: now 1786048686106454 us; error 57 us; skew 500 ppm
I20260806 20:38:06.106783 28217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:06 dist-test-slave-tm1g krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:06.109838 28217 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:38:06.111258 28217 webserver.cc:460] Webserver started at http://127.23.200.1:43255/ using document root <none> and password file <none>
I20260806 20:38:06.111919 28217 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:06.112063 28217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:06.112376 28217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:06.115237 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "6fea08f6a0854ea29c38479ce18506d7"
format_stamp: "Formatted at 2026-08-06 20:38:06 on dist-test-slave-tm1g"
server_key: "84fcfb3f69e52c68e4039e04dcdfb1ae"
server_key_iv: "375ab9eae0e608ade7eb9fc6e66a4e46"
server_key_version: "encryptionkey@0"
I20260806 20:38:06.116008 28217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "6fea08f6a0854ea29c38479ce18506d7"
format_stamp: "Formatted at 2026-08-06 20:38:06 on dist-test-slave-tm1g"
server_key: "84fcfb3f69e52c68e4039e04dcdfb1ae"
server_key_iv: "375ab9eae0e608ade7eb9fc6e66a4e46"
server_key_version: "encryptionkey@0"
I20260806 20:38:06.120854 28217 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:38:06.124162 28233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:06.125335 28217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:38:06.125521 28217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "6fea08f6a0854ea29c38479ce18506d7"
format_stamp: "Formatted at 2026-08-06 20:38:06 on dist-test-slave-tm1g"
server_key: "84fcfb3f69e52c68e4039e04dcdfb1ae"
server_key_iv: "375ab9eae0e608ade7eb9fc6e66a4e46"
server_key_version: "encryptionkey@0"
I20260806 20:38:06.125715 28217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:06.142189 28217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:06.146235 28217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:06.146549 28217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:06.147491 28217 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:06.148834 28217 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:06.148936 28217 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:06.149086 28217 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:06.149158 28217 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:06.191736 28217 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:43581
I20260806 20:38:06.191779 28353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:43581 every 8 connection(s)
I20260806 20:38:06.193063 28217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:06.200349 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28217
I20260806 20:38:06.200548 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:06.201212 24352 external_mini_cluster.cc:1428] Setting key aed6d11543cf0642ce29b42ef6f59b84
Aug 06 20:38:06 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:06.213654 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:06.195497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:37489 (local address 127.23.200.62:34243)
0806 20:38:06.195970 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:06.195975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:06.196798 (+ 823us) server_negotiation.cc:407] Connection header received
0806 20:38:06.198679 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:06.198685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:06.198772 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:06.198891 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:06.200375 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.201194 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.203268 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.203708 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.203810 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:06.207942 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:06.207948 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:06.207951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:06.207981 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:06.210639 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:06.211355 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:06.211361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:06.211369 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:06.211486 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:06.211917 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:06.211922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:06.211930 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:06.212195 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:06.212352 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:06.213052 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 20:38:06.213285 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":252,"threads_started":1}
I20260806 20:38:06.214740 28359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:06.195943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34243 (local address 127.23.200.1:37489)
0806 20:38:06.196618 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:06.196654 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:06.198409 (+ 1755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:06.199131 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:06.199140 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:06.199643 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:06.200201 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:06.200217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.201419 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.201425 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:06.203138 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:06.203151 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.203950 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.203958 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:06.204285 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:06.205014 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:06.205055 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:06.207722 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:06.210780 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:06.210789 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:06.210804 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:06.211227 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:06.211597 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:06.211603 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:06.211606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:06.211797 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:06.212324 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:06.212332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:06.212669 (+ 337us) client_negotiation.cc:769] Sending connection context
0806 20:38:06.212931 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 20:38:06.213448 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":263,"threads_started":1}
I20260806 20:38:06.216323 28354 heartbeater.cc:347] Connected to a master server at 127.23.200.62:34243
I20260806 20:38:06.216629 28354 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:06.217351 28354 heartbeater.cc:511] Master 127.23.200.62:34243 requested a full tablet report, sending...
I20260806 20:38:06.219730 28154 ts_manager.cc:194] Registered new tserver with Master: 6fea08f6a0854ea29c38479ce18506d7 (127.23.200.1:43581)
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:38:06.221724 28154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:37489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:38:06.245533 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34243
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:06.567662 28363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:06.567931 28363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:06.568027 28363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:06.575170 28363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:06.575273 28363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:06.575387 28363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:06.584863 28363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:06.586257 28363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:06.587553 28363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:06.600273 28368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:06.600219 28371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:06.600831 28369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:06.601343 28363 server_base.cc:1034] running on GCE node
I20260806 20:38:06.602039 28363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:06.603148 28363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:06.604379 28363 hybrid_clock.cc:648] HybridClock initialized: now 1786048686604319 us; error 50 us; skew 500 ppm
I20260806 20:38:06.604743 28363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:06 dist-test-slave-tm1g krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:06.608371 28363 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:38:06.611423 28363 webserver.cc:460] Webserver started at http://127.23.200.2:35159/ using document root <none> and password file <none>
I20260806 20:38:06.612134 28363 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:06.612325 28363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:06.612676 28363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:06.615633 28363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "698a5a58612c44faa5302bb144e1dd2f"
format_stamp: "Formatted at 2026-08-06 20:38:06 on dist-test-slave-tm1g"
server_key: "5e283c00ca62cdbaeb2df311b9656d00"
server_key_iv: "f6b22be1c1bfdc9f0b5d51fdbcc9050a"
server_key_version: "encryptionkey@0"
I20260806 20:38:06.616410 28363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "698a5a58612c44faa5302bb144e1dd2f"
format_stamp: "Formatted at 2026-08-06 20:38:06 on dist-test-slave-tm1g"
server_key: "5e283c00ca62cdbaeb2df311b9656d00"
server_key_iv: "f6b22be1c1bfdc9f0b5d51fdbcc9050a"
server_key_version: "encryptionkey@0"
I20260806 20:38:06.621402 28363 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:38:06.624861 28378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:06.626169 28363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:38:06.626356 28363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "698a5a58612c44faa5302bb144e1dd2f"
format_stamp: "Formatted at 2026-08-06 20:38:06 on dist-test-slave-tm1g"
server_key: "5e283c00ca62cdbaeb2df311b9656d00"
server_key_iv: "f6b22be1c1bfdc9f0b5d51fdbcc9050a"
server_key_version: "encryptionkey@0"
I20260806 20:38:06.626533 28363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:06.645187 28363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:06.649289 28363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:06.649574 28363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:06.650560 28363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:06.651878 28363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:06.651952 28363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:06.652127 28363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:06.652179 28363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:06.685982 28363 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:38817
I20260806 20:38:06.686031 28497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:38817 every 8 connection(s)
I20260806 20:38:06.687279 28363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:06.693959 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28363
I20260806 20:38:06.694180 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:06.694662 24352 external_mini_cluster.cc:1428] Setting key 7402162ae048e790c107d93b934f472a
Aug 06 20:38:06 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048686, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:06.707988 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:06.689890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:55871 (local address 127.23.200.62:34243)
0806 20:38:06.690107 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:06.690114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:06.691373 (+ 1259us) server_negotiation.cc:407] Connection header received
0806 20:38:06.693128 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:06.693133 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:06.693205 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:06.693296 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:06.694946 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.695905 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.697812 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.698639 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.698789 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:06.702105 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:06.702114 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:06.702122 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:06.702167 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:06.704834 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:06.705761 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:06.705765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:06.705767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:06.705883 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:06.706388 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:06.706391 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:06.706393 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:06.706651 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:06.706762 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:06.707586 (+ 824us) server_negotiation.cc:299] Negotiation successful
0806 20:38:06.707748 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:38:06.709576 28500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:06.690507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34243 (local address 127.23.200.2:55871)
0806 20:38:06.691202 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:06.691244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:06.692853 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:06.693494 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:06.693502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:06.694028 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:06.694759 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:06.694783 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.696083 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.696088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:06.697660 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:06.697673 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:06.698853 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:06.698858 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:06.699052 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:06.699750 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:06.699777 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:06.701879 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:06.704974 (+ 3095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:06.704985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:06.705003 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:06.705651 (+ 648us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:06.705999 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:06.706006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:06.706012 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:06.706256 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:06.706774 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:06.706785 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:06.707206 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 20:38:06.707511 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 20:38:06.708086 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":265,"threads_started":1}
I20260806 20:38:06.711133 28498 heartbeater.cc:347] Connected to a master server at 127.23.200.62:34243
I20260806 20:38:06.711481 28498 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:06.712280 28498 heartbeater.cc:511] Master 127.23.200.62:34243 requested a full tablet report, sending...
I20260806 20:38:06.713786 28154 ts_manager.cc:194] Registered new tserver with Master: 698a5a58612c44faa5302bb144e1dd2f (127.23.200.2:38817)
I20260806 20:38:06.715065 28154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:55871
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:38:06.741919 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34243
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:07.065232 28503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:07.065541 28503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:07.065698 28503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:07.072956 28503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:07.073067 28503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:07.073246 28503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:07.083361 28503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34243
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:07.084664 28503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:07.086027 28503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:07.100030 28509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:07.100204 28510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:07.100332 28513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:07.110920 28503 server_base.cc:1034] running on GCE node
I20260806 20:38:07.111620 28503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:07.112614 28503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:07.113848 28503 hybrid_clock.cc:648] HybridClock initialized: now 1786048687113803 us; error 42 us; skew 500 ppm
I20260806 20:38:07.114164 28503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:07.117401 28503 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:38:07.119030 28503 webserver.cc:460] Webserver started at http://127.23.200.3:43671/ using document root <none> and password file <none>
I20260806 20:38:07.119827 28503 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:07.119974 28503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:07.120296 28503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:07.123258 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "3078a42fd66242c7add5b1eb48831f17"
format_stamp: "Formatted at 2026-08-06 20:38:07 on dist-test-slave-tm1g"
server_key: "601ffe3e6248881909237089ab0a1d4f"
server_key_iv: "35d55fa79f2456ca57b59ecd85ce4a9f"
server_key_version: "encryptionkey@0"
I20260806 20:38:07.124040 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "3078a42fd66242c7add5b1eb48831f17"
format_stamp: "Formatted at 2026-08-06 20:38:07 on dist-test-slave-tm1g"
server_key: "601ffe3e6248881909237089ab0a1d4f"
server_key_iv: "35d55fa79f2456ca57b59ecd85ce4a9f"
server_key_version: "encryptionkey@0"
I20260806 20:38:07.129235 28503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:38:07.132992 28520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:07.134196 28503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:38:07.134418 28503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "3078a42fd66242c7add5b1eb48831f17"
format_stamp: "Formatted at 2026-08-06 20:38:07 on dist-test-slave-tm1g"
server_key: "601ffe3e6248881909237089ab0a1d4f"
server_key_iv: "35d55fa79f2456ca57b59ecd85ce4a9f"
server_key_version: "encryptionkey@0"
I20260806 20:38:07.134634 28503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:07.159504 28503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:07.163964 28503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:07.164273 28503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:07.165233 28503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:07.166669 28503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:07.166774 28503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:07.166918 28503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:07.166998 28503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:07.201890 28503 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:36111
I20260806 20:38:07.201949 28637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:36111 every 8 connection(s)
I20260806 20:38:07.203243 28503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:07.211405 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28503
I20260806 20:38:07.211596 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:07.212023 24352 external_mini_cluster.cc:1428] Setting key 4a35d4144862a23323095aa381203765
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:07.223698 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.205840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:34501 (local address 127.23.200.62:34243)
0806 20:38:07.206034 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:07.206038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:07.207258 (+ 1220us) server_negotiation.cc:407] Connection header received
0806 20:38:07.209118 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:07.209124 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:07.209218 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:07.209332 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:07.210859 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.211716 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.213600 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.214176 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.214340 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.217758 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:07.217764 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:07.217769 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:07.217814 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:07.220502 (+ 2688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.221229 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.221235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.221240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.221370 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.221817 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.221821 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.221823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.222083 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:07.222241 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:07.223117 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 20:38:07.223338 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:38:07.224769 28640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.206353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34243 (local address 127.23.200.3:34501)
0806 20:38:07.207093 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:07.207130 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:07.208838 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:07.209562 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:07.209570 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:07.210078 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:07.210689 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:07.210706 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.211877 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.211881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:07.213445 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:07.213457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.214420 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.214431 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:07.214681 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.215454 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:07.215475 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:07.217554 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:07.220650 (+ 3096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:07.220659 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:07.220673 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:07.221099 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:07.221491 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:07.221496 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:07.221500 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:07.221687 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:07.222220 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:07.222228 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:07.222678 (+ 450us) client_negotiation.cc:769] Sending connection context
0806 20:38:07.222997 (+ 319us) client_negotiation.cc:240] Negotiation successful
0806 20:38:07.223511 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":284,"threads_started":1}
I20260806 20:38:07.224763 28354 heartbeater.cc:503] Master 127.23.200.62:34243 was elected leader, sending a full tablet report...
I20260806 20:38:07.226536 28638 heartbeater.cc:347] Connected to a master server at 127.23.200.62:34243
I20260806 20:38:07.226825 28638 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:07.227594 28638 heartbeater.cc:511] Master 127.23.200.62:34243 requested a full tablet report, sending...
I20260806 20:38:07.228873 28154 ts_manager.cc:194] Registered new tserver with Master: 3078a42fd66242c7add5b1eb48831f17 (127.23.200.3:36111)
I20260806 20:38:07.229651 28154 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:34501
I20260806 20:38:07.239668 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:07.289707 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.274288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57358 (local address 127.23.200.1:43581)
0806 20:38:07.274979 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:07.274990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:07.275026 (+ 36us) server_negotiation.cc:407] Connection header received
0806 20:38:07.275090 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:07.275096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:07.275320 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:07.275502 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:07.276997 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.278073 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.279544 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.280124 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.280279 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.284405 (+ 4126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:07.284417 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:07.284432 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:07.284467 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:07.287304 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.287928 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.287935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.287937 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.288054 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.288384 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.288387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.288389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.288716 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:07.288947 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:07.289195 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 20:38:07.289409 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":302,"threads_started":1}
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:07.308634 28357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.294745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.23.200.62:34243)
0806 20:38:07.294979 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:07.294988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:07.295203 (+ 215us) server_negotiation.cc:407] Connection header received
0806 20:38:07.295415 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:07.295421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:07.295494 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:07.295600 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:07.296853 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.297601 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.298951 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.299566 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.299711 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.303751 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:07.303757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:07.303760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:07.303795 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:07.306220 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.306874 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.306877 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.306880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.307008 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.307330 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.307333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.307335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.307661 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:07.307851 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:07.308181 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 20:38:07.308389 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 20:38:07.309300 28154 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45380
I20260806 20:38:07.310381 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28217
I20260806 20:38:07.322575 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.311254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.23.200.1:43581)
0806 20:38:07.311441 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:07.311445 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:07.311689 (+ 244us) server_negotiation.cc:407] Connection header received
0806 20:38:07.311818 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:07.311822 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:07.311885 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:07.311965 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:07.313138 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.313841 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.315136 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.315596 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.315707 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.317874 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:07.317879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:07.317882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:07.317914 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:07.320457 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.321023 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.321027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.321029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.321149 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.321444 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.321447 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.321449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.321680 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:07.321865 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:07.322095 (+ 230us) server_negotiation.cc:299] Negotiation successful
0806 20:38:07.322256 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:38:07.323172 28349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:07.407176 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28217
I20260806 20:38:07.425801 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28363
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:38:07.442006 28651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.426859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58528 (local address 127.23.200.2:38817)
0806 20:38:07.427398 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:07.427405 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:07.427430 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:38:07.427499 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:07.427506 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:07.427717 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:07.427898 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:07.429127 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.430239 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.431655 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.432218 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.432356 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.436511 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:07.436521 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:07.436536 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:07.436574 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:07.439705 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.440289 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.440296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.440298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.440421 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.440725 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.440728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.440730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.441060 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:07.441286 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:07.441530 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 20:38:07.441724 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":270,"threads_started":1}
I20260806 20:38:07.443372 28493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:07.527308 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28363
I20260806 20:38:07.544803 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28503
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:38:07.564611 28653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:07.545899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.23.200.3:36111)
0806 20:38:07.546609 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:07.546616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:07.546644 (+ 28us) server_negotiation.cc:407] Connection header received
0806 20:38:07.546703 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:07.546709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:07.546925 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:07.547059 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:07.548442 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.549515 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.550811 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:07.551389 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:07.551508 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:07.559184 (+ 7676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:07.559196 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:07.559211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:07.559254 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:07.562133 (+ 2879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.562807 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.562814 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.562816 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.562945 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:07.563249 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:07.563252 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:07.563253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:07.563585 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:07.563815 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:07.564016 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 20:38:07.564211 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":417,"threads_started":1}
I20260806 20:38:07.565922 28633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:07.654870 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28503
I20260806 20:38:07.672077 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28119
I20260806 20:38:07.673346 28185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:07.781981 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28119
2026-08-06T20:38:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3937 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28658](info): set up 2 sockets
Aug 06 20:38:07 dist-test-slave-tm1g krb5kdc[28658](info): commencing operation
krb5kdc: starting...
W20260806 20:38:08.890058 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:09 dist-test-slave-tm1g krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:38:09.081480 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35467
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35467
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:09.401105 28672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:09.401392 28672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:09.401569 28672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:09.408640 28672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:09.408731 28672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:09.408776 28672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:09.408856 28672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:09.408929 28672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:09.418417 28672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:09.419625 28672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:09.420890 28672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:09.432279 28680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:09.432375 28679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:09.432401 28682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:09.433451 28672 server_base.cc:1034] running on GCE node
I20260806 20:38:09.434167 28672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:09.435510 28672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:09.436703 28672 hybrid_clock.cc:648] HybridClock initialized: now 1786048689436660 us; error 34 us; skew 500 ppm
I20260806 20:38:09.437049 28672 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:09 dist-test-slave-tm1g krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048689, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:09.445552 28672 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:38:09.446729 28672 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:09.446830 28672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:09.447187 28672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:09.450068 28672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "e249fb30d71747309e01b35f1c2b3932"
format_stamp: "Formatted at 2026-08-06 20:38:09 on dist-test-slave-tm1g"
server_key: "7b24e71958b96212370747b22c743dae"
server_key_iv: "e495e31093eb174b6440722491ddcac6"
server_key_version: "encryptionkey@0"
I20260806 20:38:09.450937 28672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "e249fb30d71747309e01b35f1c2b3932"
format_stamp: "Formatted at 2026-08-06 20:38:09 on dist-test-slave-tm1g"
server_key: "7b24e71958b96212370747b22c743dae"
server_key_iv: "e495e31093eb174b6440722491ddcac6"
server_key_version: "encryptionkey@0"
I20260806 20:38:09.455683 28672 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:38:09.458968 28688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:09.460191 28672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260806 20:38:09.460371 28672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "e249fb30d71747309e01b35f1c2b3932"
format_stamp: "Formatted at 2026-08-06 20:38:09 on dist-test-slave-tm1g"
server_key: "7b24e71958b96212370747b22c743dae"
server_key_iv: "e495e31093eb174b6440722491ddcac6"
server_key_version: "encryptionkey@0"
I20260806 20:38:09.460583 28672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:09.483455 28672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:09.487805 28672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:09.488090 28672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:09.505681 28672 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:35467
I20260806 20:38:09.505702 28740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:35467 every 8 connection(s)
I20260806 20:38:09.506582 28672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:09.507992 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28672
I20260806 20:38:09.508208 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:09.509034 24352 external_mini_cluster.cc:1428] Setting key 510ecd33729348381d2d6d98065e1784
I20260806 20:38:09.511237 28741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 20:38:09 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:09.524981 28741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932: Bootstrap starting.
I20260806 20:38:09.529585 28741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:09.530982 28741 log.cc:824] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:09.534536 28741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932: No bootstrap required, opened a new log
I20260806 20:38:09.535786 28745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:09.511100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52544 (local address 127.23.200.62:35467)
0806 20:38:09.511960 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:09.511972 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:09.512010 (+ 38us) server_negotiation.cc:407] Connection header received
0806 20:38:09.513621 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:09.513642 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:09.514000 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:09.514368 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:09.516160 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:09.517452 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:09.519217 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:09.519845 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:09.519971 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:09.525058 (+ 5087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:09.525072 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:09.525094 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:09.525143 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:09.529864 (+ 4721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:09.531219 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:09.531227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:09.531241 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:09.531403 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:09.531878 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:09.531883 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:09.531887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:09.532422 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:09.532741 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:09.533872 (+ 1131us) server_negotiation.cc:299] Negotiation successful
0806 20:38:09.534340 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":455,"threads_started":1}
I20260806 20:38:09.543908 28741 raft_consensus.cc:348] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e249fb30d71747309e01b35f1c2b3932" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35467 } }
I20260806 20:38:09.544364 28741 raft_consensus.cc:374] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:09.544471 28741 raft_consensus.cc:729] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e249fb30d71747309e01b35f1c2b3932, State: Initialized, Role: FOLLOWER
I20260806 20:38:09.545132 28741 consensus_queue.cc:260] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e249fb30d71747309e01b35f1c2b3932" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35467 } }
I20260806 20:38:09.545464 28741 raft_consensus.cc:388] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:09.545619 28741 raft_consensus.cc:482] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:09.545806 28741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:09.548667 28741 raft_consensus.cc:504] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e249fb30d71747309e01b35f1c2b3932" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35467 } }
I20260806 20:38:09.549269 28741 leader_election.cc:302] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e249fb30d71747309e01b35f1c2b3932; no voters:
I20260806 20:38:09.550014 28741 leader_election.cc:288] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:09.550165 28747 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:09.552505 28741 sys_catalog.cc:564] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:09.553099 28747 raft_consensus.cc:686] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [term 1 LEADER]: Becoming Leader. State: Replica: e249fb30d71747309e01b35f1c2b3932, State: Running, Role: LEADER
I20260806 20:38:09.553618 28747 consensus_queue.cc:237] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e249fb30d71747309e01b35f1c2b3932" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35467 } }
I20260806 20:38:09.557291 28748 sys_catalog.cc:455] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e249fb30d71747309e01b35f1c2b3932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e249fb30d71747309e01b35f1c2b3932" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35467 } } }
I20260806 20:38:09.557665 28748 sys_catalog.cc:458] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:09.558158 28749 sys_catalog.cc:455] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e249fb30d71747309e01b35f1c2b3932. Latest consensus state: current_term: 1 leader_uuid: "e249fb30d71747309e01b35f1c2b3932" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e249fb30d71747309e01b35f1c2b3932" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35467 } } }
I20260806 20:38:09.558516 28749 sys_catalog.cc:458] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:09.573329 28758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:09.577706 28758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:09.587903 28758 catalog_manager.cc:1347] Generated new cluster ID: db60f6a772784f64b9ec7b76393776e6
I20260806 20:38:09.588047 28758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:09.603464 28758 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:09.604717 28758 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:09.612125 28758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932: Generated new TSK 0
I20260806 20:38:09.612905 28758 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:38:09.659546 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35467
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:10.000716 28770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:10.001003 28770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:10.001101 28770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:10.008275 28770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:10.008387 28770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:10.008512 28770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:10.018059 28770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:10.019397 28770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:10.020659 28770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:10.033824 28776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:10.033850 28778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:10.034380 28775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:10.035159 28770 server_base.cc:1034] running on GCE node
I20260806 20:38:10.035727 28770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:10.036578 28770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:10.037788 28770 hybrid_clock.cc:648] HybridClock initialized: now 1786048690037749 us; error 31 us; skew 500 ppm
I20260806 20:38:10.038133 28770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:10 dist-test-slave-tm1g krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:10.041148 28770 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:38:10.042223 28770 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:10.042359 28770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:10.042649 28770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:10.045433 28770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "6a655d5f2ad24e39bf7f300a73ad466d"
format_stamp: "Formatted at 2026-08-06 20:38:10 on dist-test-slave-tm1g"
server_key: "dfe46c96e3ed4f41b68360973216740a"
server_key_iv: "5c9a607296ceae94e017d968c35f2144"
server_key_version: "encryptionkey@0"
I20260806 20:38:10.046170 28770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "6a655d5f2ad24e39bf7f300a73ad466d"
format_stamp: "Formatted at 2026-08-06 20:38:10 on dist-test-slave-tm1g"
server_key: "dfe46c96e3ed4f41b68360973216740a"
server_key_iv: "5c9a607296ceae94e017d968c35f2144"
server_key_version: "encryptionkey@0"
I20260806 20:38:10.050928 28770 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:38:10.054672 28784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:10.055994 28770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:38:10.056123 28770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "6a655d5f2ad24e39bf7f300a73ad466d"
format_stamp: "Formatted at 2026-08-06 20:38:10 on dist-test-slave-tm1g"
server_key: "dfe46c96e3ed4f41b68360973216740a"
server_key_iv: "5c9a607296ceae94e017d968c35f2144"
server_key_version: "encryptionkey@0"
I20260806 20:38:10.056294 28770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:10.091938 28770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:10.095808 28770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:10.096045 28770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:10.096946 28770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:10.098268 28770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:10.098413 28770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:10.098502 28770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:10.098558 28770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:10.131183 28770 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:44763
I20260806 20:38:10.131268 28910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:44763 every 8 connection(s)
I20260806 20:38:10.132066 28770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:10.138661 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28770
I20260806 20:38:10.138880 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:10.139755 24352 external_mini_cluster.cc:1428] Setting key f5ce46bcc9c7656b9ca94abd183c5e20
Aug 06 20:38:10 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:10.156097 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:10.134525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:37591 (local address 127.23.200.62:35467)
0806 20:38:10.134981 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:10.134985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:10.135788 (+ 803us) server_negotiation.cc:407] Connection header received
0806 20:38:10.137632 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:10.137638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:10.137735 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:10.137854 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:10.139968 (+ 2114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.140862 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.142818 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.143591 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.143741 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:10.147544 (+ 3803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:10.147550 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:10.147553 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:10.147583 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:10.151070 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:10.152160 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:10.152167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:10.152171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:10.152473 (+ 302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:10.153055 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:10.153063 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:10.153069 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:10.153447 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:10.153605 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:10.155111 (+ 1506us) server_negotiation.cc:299] Negotiation successful
0806 20:38:10.155617 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":273,"threads_started":1}
I20260806 20:38:10.157043 28914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:10.134965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35467 (local address 127.23.200.1:37591)
0806 20:38:10.135632 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:10.135673 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:10.137341 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:10.138073 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:10.138081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:10.138922 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:10.139768 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:10.139799 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.141028 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.141033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:10.142653 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:10.142666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.143807 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.143811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:10.143987 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:10.144657 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:10.144679 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:10.147306 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:10.151235 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:10.151246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:10.151263 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:10.151913 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:10.152615 (+ 702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:10.152623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:10.152648 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:10.152888 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:10.153658 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:10.153668 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:10.154433 (+ 765us) client_negotiation.cc:769] Sending connection context
0806 20:38:10.154980 (+ 547us) client_negotiation.cc:240] Negotiation successful
0806 20:38:10.155570 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":269,"threads_started":1}
I20260806 20:38:10.159287 28911 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35467
I20260806 20:38:10.159638 28911 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:10.160504 28911 heartbeater.cc:511] Master 127.23.200.62:35467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:38:10.163189 28706 ts_manager.cc:194] Registered new tserver with Master: 6a655d5f2ad24e39bf7f300a73ad466d (127.23.200.1:44763)
I20260806 20:38:10.164660 28706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:37591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:38:10.186960 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35467
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:10.503010 28917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:10.503307 28917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:10.503481 28917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:10.510622 28917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:10.510753 28917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:10.510928 28917 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:10.520187 28917 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:10.521349 28917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:10.522739 28917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:10.534930 28926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:10.534886 28924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:10.534988 28923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:10.535758 28917 server_base.cc:1034] running on GCE node
I20260806 20:38:10.536361 28917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:10.537207 28917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:10.538408 28917 hybrid_clock.cc:648] HybridClock initialized: now 1786048690538356 us; error 42 us; skew 500 ppm
I20260806 20:38:10.538717 28917 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:10 dist-test-slave-tm1g krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:10.541690 28917 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:38:10.542832 28917 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:10.542964 28917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:10.543270 28917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:10.545989 28917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "d7199e2b0eed4914ad5a166a9f67bd0a"
format_stamp: "Formatted at 2026-08-06 20:38:10 on dist-test-slave-tm1g"
server_key: "0993368e91672f8bca938a3aae84ed66"
server_key_iv: "b6edd6bb2fef4047630dbb609bc16524"
server_key_version: "encryptionkey@0"
I20260806 20:38:10.546869 28917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "d7199e2b0eed4914ad5a166a9f67bd0a"
format_stamp: "Formatted at 2026-08-06 20:38:10 on dist-test-slave-tm1g"
server_key: "0993368e91672f8bca938a3aae84ed66"
server_key_iv: "b6edd6bb2fef4047630dbb609bc16524"
server_key_version: "encryptionkey@0"
I20260806 20:38:10.551659 28917 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260806 20:38:10.555083 28932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:10.556231 28917 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:38:10.556434 28917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "d7199e2b0eed4914ad5a166a9f67bd0a"
format_stamp: "Formatted at 2026-08-06 20:38:10 on dist-test-slave-tm1g"
server_key: "0993368e91672f8bca938a3aae84ed66"
server_key_iv: "b6edd6bb2fef4047630dbb609bc16524"
server_key_version: "encryptionkey@0"
I20260806 20:38:10.556672 28917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:10.579084 28917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:10.583272 28917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:10.583539 28917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:10.584448 28917 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:10.585693 28917 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:10.585829 28917 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:10.585953 28917 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:10.586047 28917 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:10.618801 28917 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:38759
I20260806 20:38:10.618883 29050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:38759 every 8 connection(s)
I20260806 20:38:10.619644 28917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:10.623708 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 28917
I20260806 20:38:10.623894 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:10.624462 24352 external_mini_cluster.cc:1428] Setting key 23b91ca4bb4d05a1e0b9a01084aec74c
Aug 06 20:38:10 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048690, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:10.642457 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:10.622076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:42327 (local address 127.23.200.62:35467)
0806 20:38:10.622257 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:10.622262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:10.623451 (+ 1189us) server_negotiation.cc:407] Connection header received
0806 20:38:10.625928 (+ 2477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:10.625934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:10.626024 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:10.626130 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:10.628126 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.628827 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.631304 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.631735 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.631826 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:10.635647 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:10.635656 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:10.635664 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:10.635708 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:10.639276 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:10.639969 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:10.639978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:10.639984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:10.640156 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:10.640559 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:10.640567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:10.640573 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:10.640933 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:10.641099 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:10.641704 (+ 605us) server_negotiation.cc:299] Negotiation successful
0806 20:38:10.641977 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 20:38:10.643076 29053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:10.622565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35467 (local address 127.23.200.2:42327)
0806 20:38:10.623236 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:10.623275 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:10.625680 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:10.626396 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:10.626404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:10.627121 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:10.627917 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:10.627944 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.628993 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.629001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:10.631165 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:10.631182 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:10.631954 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:10.631963 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:10.632277 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:10.632971 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:10.633023 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:10.635414 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:10.639421 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:10.639427 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:10.639437 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:10.639828 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:10.640270 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:10.640273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:10.640275 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:10.640441 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:10.641067 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:10.641072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:10.641340 (+ 268us) client_negotiation.cc:769] Sending connection context
0806 20:38:10.641545 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 20:38:10.641890 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":262,"threads_started":1}
I20260806 20:38:10.644551 29051 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35467
I20260806 20:38:10.644846 29051 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:10.645470 29051 heartbeater.cc:511] Master 127.23.200.62:35467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
I20260806 20:38:10.646885 28706 ts_manager.cc:194] Registered new tserver with Master: d7199e2b0eed4914ad5a166a9f67bd0a (127.23.200.2:38759)
I20260806 20:38:10.647677 28706 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:42327
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:38:10.673565 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35467
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:10.988533 29056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:10.988824 29056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:10.989005 29056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:10.996049 29056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:10.996189 29056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:10.996411 29056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:11.005640 29056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35467
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:11.006870 29056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:11.008165 29056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:11.021070 29061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:11.021133 29062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:11.021037 29064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:11.022053 29056 server_base.cc:1034] running on GCE node
I20260806 20:38:11.022701 29056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:11.023530 29056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:11.024726 29056 hybrid_clock.cc:648] HybridClock initialized: now 1786048691024696 us; error 46 us; skew 500 ppm
I20260806 20:38:11.025053 29056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:11 dist-test-slave-tm1g krb5kdc[28658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048691, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:11.028158 29056 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:38:11.029284 29056 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:11.029412 29056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:11.029762 29056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:11.032626 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "73c3421c0553499799a666ced78380f6"
format_stamp: "Formatted at 2026-08-06 20:38:11 on dist-test-slave-tm1g"
server_key: "fecfd99eab0f7fbf651d63c88d6046f8"
server_key_iv: "f7bba16b4cf326bf8316e9c3716b84f5"
server_key_version: "encryptionkey@0"
I20260806 20:38:11.033406 29056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "73c3421c0553499799a666ced78380f6"
format_stamp: "Formatted at 2026-08-06 20:38:11 on dist-test-slave-tm1g"
server_key: "fecfd99eab0f7fbf651d63c88d6046f8"
server_key_iv: "f7bba16b4cf326bf8316e9c3716b84f5"
server_key_version: "encryptionkey@0"
I20260806 20:38:11.038213 29056 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260806 20:38:11.041514 29070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:11.042845 29056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:11.043013 29056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "73c3421c0553499799a666ced78380f6"
format_stamp: "Formatted at 2026-08-06 20:38:11 on dist-test-slave-tm1g"
server_key: "fecfd99eab0f7fbf651d63c88d6046f8"
server_key_iv: "f7bba16b4cf326bf8316e9c3716b84f5"
server_key_version: "encryptionkey@0"
I20260806 20:38:11.043228 29056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:11.072595 29056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:11.076864 29056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:11.077150 29056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:11.078095 29056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:11.079402 29056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:11.079540 29056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:11.079663 29056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:11.079756 29056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:11.113058 29056 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:44843
I20260806 20:38:11.113143 29187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:44843 every 8 connection(s)
I20260806 20:38:11.113945 29056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:11.121608 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29056
I20260806 20:38:11.121783 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:11.122223 24352 external_mini_cluster.cc:1428] Setting key d4e5f3b4812555954f3749e2a74a6cd2
Aug 06 20:38:11 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048691, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:11.134564 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.116465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:33635 (local address 127.23.200.62:35467)
0806 20:38:11.116667 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.116672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.117759 (+ 1087us) server_negotiation.cc:407] Connection header received
0806 20:38:11.119641 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.119646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.119736 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.119852 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:11.121472 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.122388 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.124234 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.124863 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.124987 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.128401 (+ 3414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:11.128406 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:11.128409 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:11.128441 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:11.131413 (+ 2972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.132145 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.132151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.132154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.132306 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.132756 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.132761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.132763 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.133121 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:11.133294 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.133951 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.134191 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:38:11.135300 29190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.116884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35467 (local address 127.23.200.3:33635)
0806 20:38:11.117592 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:11.117639 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:11.119379 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:11.120081 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:11.120089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:11.120577 (+ 488us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:11.121172 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.121210 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.122541 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.122547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.124090 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.124104 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.125071 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.125076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.125377 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.126076 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:11.126097 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:11.128225 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:11.131583 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:11.131590 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:11.131600 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:11.132021 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:11.132474 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:11.132478 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:11.132480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:11.132645 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:11.133239 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:11.133244 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:11.133550 (+ 306us) client_negotiation.cc:769] Sending connection context
0806 20:38:11.133801 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 20:38:11.134240 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":292,"threads_started":1}
I20260806 20:38:11.137010 29188 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35467
I20260806 20:38:11.137326 29188 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:11.138016 29188 heartbeater.cc:511] Master 127.23.200.62:35467 requested a full tablet report, sending...
I20260806 20:38:11.139436 28705 ts_manager.cc:194] Registered new tserver with Master: 73c3421c0553499799a666ced78380f6 (127.23.200.3:44843)
I20260806 20:38:11.140302 28705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:33635
I20260806 20:38:11.149665 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:38:11.166520 28913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.154282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.23.200.62:35467)
0806 20:38:11.154550 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.154558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.155103 (+ 545us) server_negotiation.cc:407] Connection header received
0806 20:38:11.155225 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.155228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.155282 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.155357 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:11.156799 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.157540 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.158791 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.159198 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.159286 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.161649 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:11.161653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:11.161656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:11.161683 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:11.164040 (+ 2357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.164734 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.164738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.164740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.164863 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.165203 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.165206 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.165208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.165520 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:11.165772 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.166014 (+ 242us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.166171 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 20:38:11.167680 28911 heartbeater.cc:503] Master 127.23.200.62:35467 was elected leader, sending a full tablet report...
I20260806 20:38:11.170590 28705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52552:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:11.177737 28705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:11.203694 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.189917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.23.200.2:38759)
0806 20:38:11.190471 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.190481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.191773 (+ 1292us) server_negotiation.cc:407] Connection header received
0806 20:38:11.192089 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.192097 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.192337 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.192579 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:11.193716 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.195894 (+ 2178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.199604 (+ 3710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.201601 (+ 1997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.201748 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.202000 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.202991 (+ 991us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.203246 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":937,"threads_started":1}
I20260806 20:38:11.204911 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.189742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:38759 (local address 127.0.0.1:59940)
0806 20:38:11.191623 (+ 1881us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:11.191661 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:11.191856 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:11.192838 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:11.192844 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:11.192873 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:11.193459 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.193477 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.196158 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.196166 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.199373 (+ 3207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.199394 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.202280 (+ 2886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.202291 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.202646 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.202722 (+ 76us) client_negotiation.cc:769] Sending connection context
0806 20:38:11.204037 (+ 1315us) client_negotiation.cc:240] Negotiation successful
0806 20:38:11.204327 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1690,"thread_start_us":1562,"threads_started":1}
I20260806 20:38:11.209301 28982 tablet_service.cc:1467] Processing CreateTablet for tablet aff9ef245daa491a8689654f136796fb (DEFAULT_TABLE table=test-table [id=fedd419d158a4c1dbfa30cb7cfafa536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:11.210691 28982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff9ef245daa491a8689654f136796fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:11.211885 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.190511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45412 (local address 127.23.200.1:44763)
0806 20:38:11.191123 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.191131 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.202875 (+ 11744us) server_negotiation.cc:407] Connection header received
0806 20:38:11.203426 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.203438 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.203738 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.203902 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:11.204913 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.206495 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.208998 (+ 2503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.210456 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.210608 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.210824 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.211391 (+ 567us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.211550 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":504,"threads_started":1}
I20260806 20:38:11.212625 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.190074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:44763 (local address 127.0.0.1:45412)
0806 20:38:11.202728 (+ 12654us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:11.202752 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:11.203178 (+ 426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:11.204199 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:11.204206 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:11.204242 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:11.204707 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.204720 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.206654 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.206662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.208829 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.208844 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.211009 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.211015 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.211177 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.211208 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 20:38:11.212081 (+ 873us) client_negotiation.cc:240] Negotiation successful
0806 20:38:11.212215 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12465,"thread_start_us":12829,"threads_started":1}
I20260806 20:38:11.213202 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.192564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:44843 (local address 127.0.0.1:47310)
0806 20:38:11.204777 (+ 12213us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:11.204795 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:11.204918 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:11.206525 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:11.206529 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:11.206559 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:11.206966 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.206975 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.208497 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.208502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.210477 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.210491 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.212444 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.212450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.212635 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.212665 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 20:38:11.212748 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 20:38:11.212883 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12078,"thread_start_us":12596,"threads_started":1}
I20260806 20:38:11.213963 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.192716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47310 (local address 127.23.200.3:44843)
0806 20:38:11.205608 (+ 12892us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.205617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.205647 (+ 30us) server_negotiation.cc:407] Connection header received
0806 20:38:11.205741 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.205750 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.205930 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.206070 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:11.207190 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.208357 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.210659 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.212218 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.212347 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.212571 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.213463 (+ 892us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.213629 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12726,"thread_start_us":12569,"threads_started":1}
I20260806 20:38:11.216024 28838 tablet_service.cc:1467] Processing CreateTablet for tablet aff9ef245daa491a8689654f136796fb (DEFAULT_TABLE table=test-table [id=fedd419d158a4c1dbfa30cb7cfafa536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:11.217038 28838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff9ef245daa491a8689654f136796fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:11.221472 29119 tablet_service.cc:1467] Processing CreateTablet for tablet aff9ef245daa491a8689654f136796fb (DEFAULT_TABLE table=test-table [id=fedd419d158a4c1dbfa30cb7cfafa536]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:11.222867 29119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aff9ef245daa491a8689654f136796fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:11.224843 29206 tablet_bootstrap.cc:492] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Bootstrap starting.
I20260806 20:38:11.228485 29206 tablet_bootstrap.cc:654] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:11.228927 29207 tablet_bootstrap.cc:492] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Bootstrap starting.
I20260806 20:38:11.230237 29206 log.cc:824] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:11.232744 29206 tablet_bootstrap.cc:492] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: No bootstrap required, opened a new log
I20260806 20:38:11.232949 29207 tablet_bootstrap.cc:654] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:11.233006 29206 ts_tablet_manager.cc:1397] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 20:38:11.234118 29207 log.cc:824] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:11.237488 29207 tablet_bootstrap.cc:492] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: No bootstrap required, opened a new log
I20260806 20:38:11.237823 29207 ts_tablet_manager.cc:1397] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 20:38:11.238814 29210 tablet_bootstrap.cc:492] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Bootstrap starting.
I20260806 20:38:11.242172 29210 tablet_bootstrap.cc:654] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:11.241880 29206 raft_consensus.cc:348] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.242337 29206 raft_consensus.cc:374] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:11.242447 29206 raft_consensus.cc:729] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7199e2b0eed4914ad5a166a9f67bd0a, State: Initialized, Role: FOLLOWER
I20260806 20:38:11.243067 29206 consensus_queue.cc:260] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.243534 29210 log.cc:824] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:11.244287 29051 heartbeater.cc:503] Master 127.23.200.62:35467 was elected leader, sending a full tablet report...
I20260806 20:38:11.245121 29206 ts_tablet_manager.cc:1428] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260806 20:38:11.246656 29207 raft_consensus.cc:348] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.247104 29207 raft_consensus.cc:374] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:11.247207 29207 raft_consensus.cc:729] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a655d5f2ad24e39bf7f300a73ad466d, State: Initialized, Role: FOLLOWER
I20260806 20:38:11.247819 29207 consensus_queue.cc:260] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.248375 29210 tablet_bootstrap.cc:492] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: No bootstrap required, opened a new log
I20260806 20:38:11.248678 29210 ts_tablet_manager.cc:1397] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 20:38:11.250659 29207 ts_tablet_manager.cc:1428] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Time spent starting tablet: real 0.013s user 0.004s sys 0.008s
I20260806 20:38:11.256294 29211 raft_consensus.cc:482] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:11.256561 29211 raft_consensus.cc:504] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.257565 29210 raft_consensus.cc:348] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.257864 29210 raft_consensus.cc:374] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:11.257936 29210 raft_consensus.cc:729] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73c3421c0553499799a666ced78380f6, State: Initialized, Role: FOLLOWER
I20260806 20:38:11.258483 29211 leader_election.cc:288] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a655d5f2ad24e39bf7f300a73ad466d (127.23.200.1:44763), 73c3421c0553499799a666ced78380f6 (127.23.200.3:44843)
I20260806 20:38:11.258530 29210 consensus_queue.cc:260] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.259675 29188 heartbeater.cc:503] Master 127.23.200.62:35467 was elected leader, sending a full tablet report...
I20260806 20:38:11.260542 29210 ts_tablet_manager.cc:1428] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260806 20:38:11.268185 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.259280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:50133 (local address 127.23.200.1:44763)
0806 20:38:11.259468 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.259476 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.261592 (+ 2116us) server_negotiation.cc:407] Connection header received
0806 20:38:11.261773 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.261777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.261829 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.261930 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:11.262934 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.263828 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.266249 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.267349 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.267500 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.267576 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.267805 (+ 229us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.267943 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:38:11.268275 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.258815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:44763 (local address 127.23.200.2:50133)
0806 20:38:11.261466 (+ 2651us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:11.261491 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:11.261648 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:11.261955 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:11.261959 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:11.262334 (+ 375us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:11.262730 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.262744 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.263961 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.263968 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.266102 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.266118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.267537 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.267541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.267643 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.267666 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:38:11.267738 (+ 72us) client_negotiation.cc:240] Negotiation successful
0806 20:38:11.267841 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2510,"mutex_wait_us":1,"thread_start_us":2331,"threads_started":1}
I20260806 20:38:11.268813 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.258926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:44843 (local address 127.23.200.2:52811)
0806 20:38:11.262217 (+ 3291us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:11.262233 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:11.262409 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:11.262745 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:11.262749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:11.262774 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:11.263169 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.263180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.264539 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.264544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.266584 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:11.266594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.268096 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.268102 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:11.268327 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.268365 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:38:11.268464 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 20:38:11.268601 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3164,"thread_start_us":3063,"threads_started":1}
I20260806 20:38:11.269152 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.259359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:52811 (local address 127.23.200.3:44843)
0806 20:38:11.259556 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.259562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.262371 (+ 2809us) server_negotiation.cc:407] Connection header received
0806 20:38:11.262549 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.262554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.262620 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.262730 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:11.263343 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.264397 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.266715 (+ 2318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.267892 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.268034 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.268100 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.268760 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.268918 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 20:38:11.269162 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff9ef245daa491a8689654f136796fb" candidate_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" is_pre_election: true
I20260806 20:38:11.269742 28862 raft_consensus.cc:2454] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7199e2b0eed4914ad5a166a9f67bd0a in term 0.
I20260806 20:38:11.270030 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff9ef245daa491a8689654f136796fb" candidate_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c3421c0553499799a666ced78380f6" is_pre_election: true
I20260806 20:38:11.270536 28936 leader_election.cc:302] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a655d5f2ad24e39bf7f300a73ad466d, d7199e2b0eed4914ad5a166a9f67bd0a; no voters:
I20260806 20:38:11.270539 29142 raft_consensus.cc:2454] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7199e2b0eed4914ad5a166a9f67bd0a in term 0.
I20260806 20:38:11.270879 29211 raft_consensus.cc:2781] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:11.271041 29211 raft_consensus.cc:482] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:11.271173 29211 raft_consensus.cc:3037] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:11.273269 29211 raft_consensus.cc:504] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.273845 29211 leader_election.cc:288] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [CANDIDATE]: Term 1 election: Requested vote from peers 6a655d5f2ad24e39bf7f300a73ad466d (127.23.200.1:44763), 73c3421c0553499799a666ced78380f6 (127.23.200.3:44843)
I20260806 20:38:11.274156 28862 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff9ef245daa491a8689654f136796fb" candidate_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a655d5f2ad24e39bf7f300a73ad466d"
I20260806 20:38:11.274376 28862 raft_consensus.cc:3037] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:11.274348 29142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aff9ef245daa491a8689654f136796fb" candidate_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "73c3421c0553499799a666ced78380f6"
I20260806 20:38:11.274544 29142 raft_consensus.cc:3037] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:11.276521 28862 raft_consensus.cc:2454] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7199e2b0eed4914ad5a166a9f67bd0a in term 1.
I20260806 20:38:11.276600 29142 raft_consensus.cc:2454] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7199e2b0eed4914ad5a166a9f67bd0a in term 1.
I20260806 20:38:11.276899 28936 leader_election.cc:302] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a655d5f2ad24e39bf7f300a73ad466d, d7199e2b0eed4914ad5a166a9f67bd0a; no voters:
I20260806 20:38:11.277195 29211 raft_consensus.cc:2781] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:11.277604 29211 raft_consensus.cc:686] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 1 LEADER]: Becoming Leader. State: Replica: d7199e2b0eed4914ad5a166a9f67bd0a, State: Running, Role: LEADER
I20260806 20:38:11.278054 29211 consensus_queue.cc:237] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } }
I20260806 20:38:11.282578 28705 catalog_manager.cc:5495] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a reported cstate change: term changed from 0 to 1, leader changed from <none> to d7199e2b0eed4914ad5a166a9f67bd0a (127.23.200.2). New cstate: current_term: 1 leader_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7199e2b0eed4914ad5a166a9f67bd0a" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38759 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:11.297113 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.291988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59952 (local address 127.23.200.2:38759)
0806 20:38:11.292147 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.292152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.292357 (+ 205us) server_negotiation.cc:407] Connection header received
0806 20:38:11.292518 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.292523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.292593 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.292708 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:11.293140 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.293818 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.295180 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.295564 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.295653 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.296056 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:11.296521 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:11.296656 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.296733 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.296794 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:38:11.305053 29142 raft_consensus.cc:1261] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 1 FOLLOWER]: Refusing update from remote peer d7199e2b0eed4914ad5a166a9f67bd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:11.305050 28862 raft_consensus.cc:1261] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 1 FOLLOWER]: Refusing update from remote peer d7199e2b0eed4914ad5a166a9f67bd0a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:11.306044 29217 consensus_queue.cc:1035] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "73c3421c0553499799a666ced78380f6" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:11.306406 29211 consensus_queue.cc:1035] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a655d5f2ad24e39bf7f300a73ad466d" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 44763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:11.330039 28705 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:11.330250 28705 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52552:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:11.333877 28705 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e249fb30d71747309e01b35f1c2b3932: Sending DeleteTablet for 3 replicas of tablet aff9ef245daa491a8689654f136796fb
I20260806 20:38:11.335012 28838 tablet_service.cc:1514] Processing DeleteTablet for tablet aff9ef245daa491a8689654f136796fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:11 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:45412
I20260806 20:38:11.335037 28982 tablet_service.cc:1514] Processing DeleteTablet for tablet aff9ef245daa491a8689654f136796fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:11 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:59940
I20260806 20:38:11.335099 29119 tablet_service.cc:1514] Processing DeleteTablet for tablet aff9ef245daa491a8689654f136796fb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:11 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:47310
I20260806 20:38:11.336077 29230 tablet_replica.cc:330] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: stopping tablet replica
I20260806 20:38:11.336565 29230 raft_consensus.cc:2229] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:11.337113 29230 raft_consensus.cc:2258] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:11.337623 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28770
I20260806 20:38:11.341903 29230 ts_tablet_manager.cc:1905] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:11.349869 29230 ts_tablet_manager.cc:1918] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:11.350278 29230 log.cc:1196] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/aff9ef245daa491a8689654f136796fb
Aug 06 20:38:11 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:11.351009 29230 ts_tablet_manager.cc:1939] T aff9ef245daa491a8689654f136796fb P 6a655d5f2ad24e39bf7f300a73ad466d: Deleting consensus metadata
I20260806 20:38:11.353504 28692 catalog_manager.cc:4841] TS 6a655d5f2ad24e39bf7f300a73ad466d (127.23.200.1:44763): tablet aff9ef245daa491a8689654f136796fb (table test-table [id=fedd419d158a4c1dbfa30cb7cfafa536]) successfully deleted
I20260806 20:38:11.354836 29232 tablet_replica.cc:330] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: stopping tablet replica
I20260806 20:38:11.355391 29232 raft_consensus.cc:2229] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:11.356043 29232 raft_consensus.cc:2258] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:11.356969 29231 tablet_replica.cc:330] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: stopping tablet replica
I20260806 20:38:11.357434 29231 raft_consensus.cc:2229] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:11.358120 29231 raft_consensus.cc:2258] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:11.359480 29203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.338560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.23.200.1:44763)
0806 20:38:11.338733 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.338741 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.340363 (+ 1622us) server_negotiation.cc:407] Connection header received
0806 20:38:11.340569 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.340576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.340653 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.340744 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:11.342543 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.343701 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.345536 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.346209 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.346351 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.351637 (+ 5286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:11.351649 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:11.351665 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:11.351714 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:11.355789 (+ 4075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.356788 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.356799 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.356804 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.356977 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.357471 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.357477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.357479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.357936 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:11.358234 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.358870 (+ 636us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.359100 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 20:38:11.359946 29232 ts_tablet_manager.cc:1905] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:11.360627 29231 ts_tablet_manager.cc:1905] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:11.361091 28906 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:11.368062 29231 ts_tablet_manager.cc:1918] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:11.368284 29231 log.cc:1196] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/aff9ef245daa491a8689654f136796fb
I20260806 20:38:11.368934 29231 ts_tablet_manager.cc:1939] T aff9ef245daa491a8689654f136796fb P d7199e2b0eed4914ad5a166a9f67bd0a: Deleting consensus metadata
I20260806 20:38:11.369386 29232 ts_tablet_manager.cc:1918] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:11.369565 29232 log.cc:1196] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/aff9ef245daa491a8689654f136796fb
I20260806 20:38:11.370195 29232 ts_tablet_manager.cc:1939] T aff9ef245daa491a8689654f136796fb P 73c3421c0553499799a666ced78380f6: Deleting consensus metadata
I20260806 20:38:11.370940 28689 catalog_manager.cc:4841] TS d7199e2b0eed4914ad5a166a9f67bd0a (127.23.200.2:38759): tablet aff9ef245daa491a8689654f136796fb (table test-table [id=fedd419d158a4c1dbfa30cb7cfafa536]) successfully deleted
I20260806 20:38:11.372373 28689 catalog_manager.cc:4841] TS 73c3421c0553499799a666ced78380f6 (127.23.200.3:44843): tablet aff9ef245daa491a8689654f136796fb (table test-table [id=fedd419d158a4c1dbfa30cb7cfafa536]) successfully deleted
I20260806 20:38:11.504053 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28770
I20260806 20:38:11.528857 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28917
Aug 06 20:38:11 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:38:11.548283 29201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.529773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59968 (local address 127.23.200.2:38759)
0806 20:38:11.529993 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.530001 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.530097 (+ 96us) server_negotiation.cc:407] Connection header received
0806 20:38:11.530282 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.530288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.530473 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.530580 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:11.532750 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.534048 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.535893 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.536580 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.536707 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.542096 (+ 5389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:11.542109 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:11.542125 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:11.542170 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:11.545184 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.545999 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.546008 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.546011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.546180 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.546683 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.546687 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.546689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.547104 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:11.547387 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.547679 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.547888 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:38:11.549049 29046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:11.662922 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28917
I20260806 20:38:11.682981 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29056
Aug 06 20:38:11 dist-test-slave-tm1g krb5kdc[28658](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048689, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:38:11.697531 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:11.683726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47320 (local address 127.23.200.3:44843)
0806 20:38:11.683890 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:11.683894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:11.683913 (+ 19us) server_negotiation.cc:407] Connection header received
0806 20:38:11.684064 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:11.684069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:11.684129 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:11.684214 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:11.685462 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.686197 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.687467 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:11.687855 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:11.687946 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:11.692307 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:11.692320 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:11.692335 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:11.692370 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:11.695160 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.695819 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.695825 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.695828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.695955 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:11.696291 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:11.696294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:11.696296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:11.696630 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:11.696829 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:11.697116 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 20:38:11.697282 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:38:11.698167 29183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:11.791066 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29056
I20260806 20:38:11.809985 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 28672
I20260806 20:38:11.811336 28736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:11.922155 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 28672
2026-08-06T20:38:11Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4140 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T20:38:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:11Z Disabled control of system clock
I20260806 20:38:11.991600 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:34725
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 20:38:12.311492 29241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:12.311775 29241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:12.311923 29241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:12.319089 29241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:12.319183 29241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:12.319271 29241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:12.319375 29241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:12.319445 29241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:12.328509 29241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:34725
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:12.329797 29241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:12.331564 29241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:12.343364 29246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:12.343451 29247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:12.343447 29249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:12.344416 29241 server_base.cc:1034] running on GCE node
I20260806 20:38:12.345166 29241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:12.346570 29241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:12.347805 29241 hybrid_clock.cc:648] HybridClock initialized: now 1786048692347768 us; error 49 us; skew 500 ppm
I20260806 20:38:12.348157 29241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:12.350359 29241 webserver.cc:460] Webserver started at http://127.23.200.62:38791/ using document root <none> and password file <none>
I20260806 20:38:12.351063 29241 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:12.351192 29241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:12.351526 29241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:12.354346 29241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "2404fb334e604d62a4cf34424dd98c84"
format_stamp: "Formatted at 2026-08-06 20:38:12 on dist-test-slave-tm1g"
server_key: "47b1edc116e1cf322656b3aef20aa85a"
server_key_iv: "443d0935ad5c1aca825612f723c95a21"
server_key_version: "encryptionkey@0"
I20260806 20:38:12.355180 29241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "2404fb334e604d62a4cf34424dd98c84"
format_stamp: "Formatted at 2026-08-06 20:38:12 on dist-test-slave-tm1g"
server_key: "47b1edc116e1cf322656b3aef20aa85a"
server_key_iv: "443d0935ad5c1aca825612f723c95a21"
server_key_version: "encryptionkey@0"
I20260806 20:38:12.360134 29241 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260806 20:38:12.363437 29255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:12.364713 29241 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:38:12.364892 29241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "2404fb334e604d62a4cf34424dd98c84"
format_stamp: "Formatted at 2026-08-06 20:38:12 on dist-test-slave-tm1g"
server_key: "47b1edc116e1cf322656b3aef20aa85a"
server_key_iv: "443d0935ad5c1aca825612f723c95a21"
server_key_version: "encryptionkey@0"
I20260806 20:38:12.365114 29241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:12.370683 29241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:12.371428 29241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:12.371728 29241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:12.388546 29241 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:34725
I20260806 20:38:12.388568 29310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:34725 every 8 connection(s)
I20260806 20:38:12.389884 29241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:12.393606 29311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:12.398206 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29241
I20260806 20:38:12.398478 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:12.399135 24352 external_mini_cluster.cc:1428] Setting key 6d9bc7eb3ccbe5180c7c9984d8208270
I20260806 20:38:12.404188 29311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84: Bootstrap starting.
I20260806 20:38:12.407611 29314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:12.401296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51346 (local address 127.23.200.62:34725)
0806 20:38:12.402503 (+ 1207us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:12.402513 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:12.402554 (+ 41us) server_negotiation.cc:407] Connection header received
0806 20:38:12.404137 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:12.404157 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:12.404574 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:12.404878 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:12.406065 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:12.406072 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:12.406083 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:12.406098 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:12.406154 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:12.406169 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:12.406404 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:12.406504 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 20:38:12.406742 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":735,"thread_start_us":410,"threads_started":1}
I20260806 20:38:12.407770 29311 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:12.409092 29311 log.cc:824] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:12.411963 29311 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84: No bootstrap required, opened a new log
I20260806 20:38:12.419373 29311 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404fb334e604d62a4cf34424dd98c84" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34725 } }
I20260806 20:38:12.419672 29311 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:12.419734 29311 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2404fb334e604d62a4cf34424dd98c84, State: Initialized, Role: FOLLOWER
I20260806 20:38:12.420286 29311 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404fb334e604d62a4cf34424dd98c84" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34725 } }
I20260806 20:38:12.420482 29311 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:12.420576 29311 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:12.420697 29311 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:12.422792 29311 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404fb334e604d62a4cf34424dd98c84" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34725 } }
I20260806 20:38:12.423216 29311 leader_election.cc:302] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2404fb334e604d62a4cf34424dd98c84; no voters:
I20260806 20:38:12.423790 29311 leader_election.cc:288] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:12.423959 29316 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:12.424748 29316 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [term 1 LEADER]: Becoming Leader. State: Replica: 2404fb334e604d62a4cf34424dd98c84, State: Running, Role: LEADER
I20260806 20:38:12.425283 29316 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404fb334e604d62a4cf34424dd98c84" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34725 } }
I20260806 20:38:12.425938 29311 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:12.428401 29318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2404fb334e604d62a4cf34424dd98c84. Latest consensus state: current_term: 1 leader_uuid: "2404fb334e604d62a4cf34424dd98c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404fb334e604d62a4cf34424dd98c84" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34725 } } }
I20260806 20:38:12.428879 29318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:12.431123 29325 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:12.431031 29317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2404fb334e604d62a4cf34424dd98c84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2404fb334e604d62a4cf34424dd98c84" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34725 } } }
I20260806 20:38:12.431439 29317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:12.435530 29325 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:12.444245 29325 catalog_manager.cc:1347] Generated new cluster ID: cc50644ff4484c6c814ace8f1aaa8984
I20260806 20:38:12.444353 29325 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:12.472687 29325 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:12.473120 29325 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:12.478291 29325 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84: Generated new TSK 0
I20260806 20:38:12.479121 29325 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:38:12.493398 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34725
--builtin_ntp_servers=127.23.200.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 20:38:12.818123 29335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:12.818392 29335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:12.818560 29335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:12.825686 29335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:12.825826 29335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:12.826006 29335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:12.835191 29335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:12.836486 29335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:12.838284 29335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:12.851290 29340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:12.851265 29341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:12.851336 29343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:12.852641 29335 server_base.cc:1034] running on GCE node
I20260806 20:38:12.853211 29335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:12.854102 29335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:12.855337 29335 hybrid_clock.cc:648] HybridClock initialized: now 1786048692855276 us; error 58 us; skew 500 ppm
I20260806 20:38:12.855639 29335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:12.860930 29335 webserver.cc:460] Webserver started at http://127.23.200.1:33503/ using document root <none> and password file <none>
I20260806 20:38:12.861621 29335 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:12.861750 29335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:12.862094 29335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:12.864827 29335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "90fcf2d4f82f4d978bd2c069b78900d4"
format_stamp: "Formatted at 2026-08-06 20:38:12 on dist-test-slave-tm1g"
server_key: "0dcb130241bc72199aa597f39c17cb26"
server_key_iv: "aec61f3fe359569ad3eb786e1974e689"
server_key_version: "encryptionkey@0"
I20260806 20:38:12.865640 29335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "90fcf2d4f82f4d978bd2c069b78900d4"
format_stamp: "Formatted at 2026-08-06 20:38:12 on dist-test-slave-tm1g"
server_key: "0dcb130241bc72199aa597f39c17cb26"
server_key_iv: "aec61f3fe359569ad3eb786e1974e689"
server_key_version: "encryptionkey@0"
I20260806 20:38:12.871062 29335 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260806 20:38:12.874372 29349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:12.875756 29335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:38:12.875975 29335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "90fcf2d4f82f4d978bd2c069b78900d4"
format_stamp: "Formatted at 2026-08-06 20:38:12 on dist-test-slave-tm1g"
server_key: "0dcb130241bc72199aa597f39c17cb26"
server_key_iv: "aec61f3fe359569ad3eb786e1974e689"
server_key_version: "encryptionkey@0"
I20260806 20:38:12.876201 29335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:12.881721 29335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:12.882563 29335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:12.882846 29335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:12.883746 29335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:12.885093 29335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:12.885222 29335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:12.885362 29335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:12.885458 29335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:12.919341 29335 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:45221
I20260806 20:38:12.919427 29470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:45221 every 8 connection(s)
I20260806 20:38:12.920696 29335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:12.922426 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29335
I20260806 20:38:12.922647 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:12.923357 24352 external_mini_cluster.cc:1428] Setting key 27e139286b965833b08fbdd9b63de10c
I20260806 20:38:12.926764 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34725
--builtin_ntp_servers=127.23.200.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 20:38:12.930392 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:12.923586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:51239 (local address 127.23.200.62:34725)
0806 20:38:12.924087 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:12.924095 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:12.925430 (+ 1335us) server_negotiation.cc:407] Connection header received
0806 20:38:12.927796 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:12.927800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:12.927855 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:12.927947 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:12.929417 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:12.929425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:12.929433 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:12.929455 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:12.929511 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:12.929529 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:12.929657 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:12.929911 (+ 254us) server_negotiation.cc:299] Negotiation successful
0806 20:38:12.929975 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":247,"threads_started":1}
I20260806 20:38:12.931283 29474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:12.924198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34725 (local address 127.23.200.1:51239)
0806 20:38:12.925107 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:12.925152 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:12.927511 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:12.928211 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:12.928225 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:12.928344 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:12.929040 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:12.929046 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:12.929095 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:12.929206 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:12.929667 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:12.929686 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:12.929693 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 20:38:12.929961 (+ 268us) client_negotiation.cc:240] Negotiation successful
0806 20:38:12.930160 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":328,"threads_started":1}
I20260806 20:38:12.932960 29471 heartbeater.cc:347] Connected to a master server at 127.23.200.62:34725
I20260806 20:38:12.933259 29471 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:12.933915 29471 heartbeater.cc:511] Master 127.23.200.62:34725 requested a full tablet report, sending...
I20260806 20:38:12.935952 29273 ts_manager.cc:194] Registered new tserver with Master: 90fcf2d4f82f4d978bd2c069b78900d4 (127.23.200.1:45221)
W20260806 20:38:13.250074 29475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:13.250399 29475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:13.250568 29475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:13.257815 29475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:13.257905 29475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:13.258071 29475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:13.266808 29475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:13.268036 29475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:13.269863 29475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:13.282553 29480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:13.282596 29483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:13.282640 29481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:13.287770 29475 server_base.cc:1034] running on GCE node
I20260806 20:38:13.288352 29475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:13.289258 29475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:13.290421 29475 hybrid_clock.cc:648] HybridClock initialized: now 1786048693290369 us; error 32 us; skew 500 ppm
I20260806 20:38:13.290696 29475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:13.293071 29475 webserver.cc:460] Webserver started at http://127.23.200.2:38941/ using document root <none> and password file <none>
I20260806 20:38:13.293723 29475 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:13.293809 29475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:13.294098 29475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:13.296932 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "a88f7e0a628a45818fc33325fad2d205"
format_stamp: "Formatted at 2026-08-06 20:38:13 on dist-test-slave-tm1g"
server_key: "08b29743f6b3fd7c61e9b6f951182fc7"
server_key_iv: "617ce6c3e0f40775a329dba5b513b83f"
server_key_version: "encryptionkey@0"
I20260806 20:38:13.297672 29475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "a88f7e0a628a45818fc33325fad2d205"
format_stamp: "Formatted at 2026-08-06 20:38:13 on dist-test-slave-tm1g"
server_key: "08b29743f6b3fd7c61e9b6f951182fc7"
server_key_iv: "617ce6c3e0f40775a329dba5b513b83f"
server_key_version: "encryptionkey@0"
I20260806 20:38:13.302743 29475 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:38:13.305939 29490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:13.307235 29475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:13.307401 29475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "a88f7e0a628a45818fc33325fad2d205"
format_stamp: "Formatted at 2026-08-06 20:38:13 on dist-test-slave-tm1g"
server_key: "08b29743f6b3fd7c61e9b6f951182fc7"
server_key_iv: "617ce6c3e0f40775a329dba5b513b83f"
server_key_version: "encryptionkey@0"
I20260806 20:38:13.307627 29475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:13.313251 29475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:13.314003 29475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:13.314262 29475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:13.315151 29475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:13.316500 29475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:13.316623 29475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:13.316759 29475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:13.316848 29475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:13.349684 29475 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:38019
I20260806 20:38:13.349732 29610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:38019 every 8 connection(s)
I20260806 20:38:13.351017 29475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:13.358528 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29475
I20260806 20:38:13.358748 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:13.359140 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.353643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:41473 (local address 127.23.200.62:34725)
0806 20:38:13.353827 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:13.353832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:13.355086 (+ 1254us) server_negotiation.cc:407] Connection header received
0806 20:38:13.356769 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:13.356775 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:13.356867 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:13.356980 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:13.358286 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:13.358291 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:13.358294 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:13.358363 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:13.358396 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:13.358410 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:13.358532 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:13.358825 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 20:38:13.358897 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 20:38:13.359324 24352 external_mini_cluster.cc:1428] Setting key 2298bd69dc99d7564bc39cd37b3205ed
I20260806 20:38:13.360276 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.353991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34725 (local address 127.23.200.2:41473)
0806 20:38:13.354934 (+ 943us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:13.354969 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:13.356497 (+ 1528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:13.357203 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:13.357210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:13.357295 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:13.357929 (+ 634us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:13.357931 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:13.357968 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:13.358082 (+ 114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:13.358518 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:13.358529 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:13.358537 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 20:38:13.358889 (+ 352us) client_negotiation.cc:240] Negotiation successful
0806 20:38:13.359137 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":334,"threads_started":1}
I20260806 20:38:13.361658 29611 heartbeater.cc:347] Connected to a master server at 127.23.200.62:34725
I20260806 20:38:13.361954 29611 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:13.362390 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34725
--builtin_ntp_servers=127.23.200.20:46447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 20:38:13.362636 29611 heartbeater.cc:511] Master 127.23.200.62:34725 requested a full tablet report, sending...
I20260806 20:38:13.364003 29273 ts_manager.cc:194] Registered new tserver with Master: a88f7e0a628a45818fc33325fad2d205 (127.23.200.2:38019)
W20260806 20:38:13.686269 29616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:13.686596 29616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:13.686687 29616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:13.693748 29616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:13.693840 29616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:13.693948 29616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:13.703320 29616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:34725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:13.704507 29616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:13.706225 29616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:13.719738 29623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:13.719728 29625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:13.720295 29622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:13.720613 29616 server_base.cc:1034] running on GCE node
I20260806 20:38:13.721233 29616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:13.722133 29616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:13.723356 29616 hybrid_clock.cc:648] HybridClock initialized: now 1786048693723312 us; error 57 us; skew 500 ppm
I20260806 20:38:13.723671 29616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:13.725939 29616 webserver.cc:460] Webserver started at http://127.23.200.3:40623/ using document root <none> and password file <none>
I20260806 20:38:13.726711 29616 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:13.726833 29616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:13.727183 29616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:13.730005 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "679457c466aa4793bf2a602771443834"
format_stamp: "Formatted at 2026-08-06 20:38:13 on dist-test-slave-tm1g"
server_key: "f330d08a0d072427fb256f49ba7ed61d"
server_key_iv: "b3267115dc7914c175b07a9e6565a4c8"
server_key_version: "encryptionkey@0"
I20260806 20:38:13.730777 29616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "679457c466aa4793bf2a602771443834"
format_stamp: "Formatted at 2026-08-06 20:38:13 on dist-test-slave-tm1g"
server_key: "f330d08a0d072427fb256f49ba7ed61d"
server_key_iv: "b3267115dc7914c175b07a9e6565a4c8"
server_key_version: "encryptionkey@0"
I20260806 20:38:13.735641 29616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 20:38:13.738888 29631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:13.740463 29616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:38:13.740653 29616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "679457c466aa4793bf2a602771443834"
format_stamp: "Formatted at 2026-08-06 20:38:13 on dist-test-slave-tm1g"
server_key: "f330d08a0d072427fb256f49ba7ed61d"
server_key_iv: "b3267115dc7914c175b07a9e6565a4c8"
server_key_version: "encryptionkey@0"
I20260806 20:38:13.740871 29616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:13.748351 29616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:13.749253 29616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:13.749583 29616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:13.750664 29616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:13.752074 29616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:13.752197 29616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:13.752323 29616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:13.752408 29616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:13.786018 29616 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:45405
I20260806 20:38:13.786069 29750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:45405 every 8 connection(s)
I20260806 20:38:13.787388 29616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:13.792142 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29616
I20260806 20:38:13.792366 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:13.792903 24352 external_mini_cluster.cc:1428] Setting key d91afaa0272d0e0dd10f45639054fc37
I20260806 20:38:13.796175 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.790005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:39837 (local address 127.23.200.62:34725)
0806 20:38:13.790186 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:13.790191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:13.791513 (+ 1322us) server_negotiation.cc:407] Connection header received
0806 20:38:13.793926 (+ 2413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:13.793932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:13.794014 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:13.794126 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:13.795422 (+ 1296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:13.795428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:13.795433 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:13.795451 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:13.795493 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:13.795505 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:13.795626 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:13.795827 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 20:38:13.795906 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:38:13.797024 29753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.790466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34725 (local address 127.23.200.3:39837)
0806 20:38:13.791347 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:13.791389 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:13.793651 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:13.794386 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:13.794397 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:13.794495 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:13.795145 (+ 650us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:13.795147 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:13.795183 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:13.795262 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:13.795612 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:13.795622 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:13.795630 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 20:38:13.795908 (+ 278us) client_negotiation.cc:240] Negotiation successful
0806 20:38:13.796148 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":274,"threads_started":1}
I20260806 20:38:13.798255 29751 heartbeater.cc:347] Connected to a master server at 127.23.200.62:34725
I20260806 20:38:13.798609 29751 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:13.799182 29751 heartbeater.cc:511] Master 127.23.200.62:34725 requested a full tablet report, sending...
I20260806 20:38:13.800503 29273 ts_manager.cc:194] Registered new tserver with Master: 679457c466aa4793bf2a602771443834 (127.23.200.3:45405)
I20260806 20:38:13.808262 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:38:13.815156 29473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.812780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51354 (local address 127.23.200.62:34725)
0806 20:38:13.812936 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:13.812940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:13.813510 (+ 570us) server_negotiation.cc:407] Connection header received
0806 20:38:13.813652 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:13.813655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:13.813703 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:13.813781 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:13.814671 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:13.814676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:13.814679 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:13.814694 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:13.814725 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:13.814737 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:13.814840 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:13.814912 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 20:38:13.814954 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:38:13.818843 29273 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51354:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:13.825568 29273 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:13.843659 29764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.837077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33854)
0806 20:38:13.838820 (+ 1743us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:13.838868 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:13.838999 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:13.840580 (+ 1581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:13.840588 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:13.840682 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:13.841551 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:13.841556 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:13.841588 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:13.841677 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:13.842864 (+ 1187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:13.842878 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:13.842887 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 20:38:13.843069 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 20:38:13.843171 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"thread_start_us":1366,"threads_started":1}
I20260806 20:38:13.844568 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.837394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:38019 (local address 127.0.0.1:46558)
0806 20:38:13.839935 (+ 2541us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:13.839952 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:13.840067 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:13.842576 (+ 2509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:13.842581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:13.842605 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:13.843254 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:13.843257 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:13.843272 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:13.843322 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:13.843886 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:13.843892 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:13.843899 (+ 7us) client_negotiation.cc:769] Sending connection context
0806 20:38:13.844068 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 20:38:13.844128 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2397,"thread_start_us":2268,"threads_started":1}
I20260806 20:38:13.844684 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.837312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33854 (local address 127.23.200.1:45221)
0806 20:38:13.840007 (+ 2695us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:13.840016 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:13.840046 (+ 30us) server_negotiation.cc:407] Connection header received
0806 20:38:13.840118 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:13.840128 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:13.840361 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:13.841139 (+ 778us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:13.842576 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:13.842585 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:13.842597 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:13.842621 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:13.842683 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:13.842706 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:13.844158 (+ 1452us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:13.844234 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 20:38:13.844334 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2516,"thread_start_us":2386,"threads_started":1}
I20260806 20:38:13.845366 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.837952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.23.200.2:38019)
0806 20:38:13.839501 (+ 1549us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:13.839509 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:13.840177 (+ 668us) server_negotiation.cc:407] Connection header received
0806 20:38:13.841254 (+ 1077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:13.841262 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:13.841465 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:13.841608 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:13.843496 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:13.843507 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:13.843520 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:13.843554 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:13.843615 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:13.843638 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:13.843827 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:13.844921 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0806 20:38:13.845028 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1384,"thread_start_us":1247,"threads_started":1}
I20260806 20:38:13.847743 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.840485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36584 (local address 127.23.200.3:45405)
0806 20:38:13.844101 (+ 3616us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:13.844110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:13.844387 (+ 277us) server_negotiation.cc:407] Connection header received
0806 20:38:13.844460 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:13.844468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:13.844653 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:13.844807 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:13.846702 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:13.846710 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:13.846721 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:13.846738 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:13.846787 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:13.846806 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:13.847071 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:13.847294 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 20:38:13.847383 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3204,"thread_start_us":3025,"threads_started":1}
I20260806 20:38:13.847793 29769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:13.840090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:45405 (local address 127.0.0.1:36584)
0806 20:38:13.844118 (+ 4028us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:13.844136 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:13.844248 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:13.845185 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:13.845190 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:13.845214 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:13.845917 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:13.845923 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:13.845938 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:13.845983 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:13.847123 (+ 1140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:13.847131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:13.847137 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 20:38:13.847289 (+ 152us) client_negotiation.cc:240] Negotiation successful
0806 20:38:13.847352 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3902,"thread_start_us":3101,"threads_started":1}
I20260806 20:38:13.848495 29401 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0c7a871f524dccb9453c4dbc604231 (DEFAULT_TABLE table=test-table [id=272b367258134acfb4e01dd71cfadd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:13.849104 29541 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0c7a871f524dccb9453c4dbc604231 (DEFAULT_TABLE table=test-table [id=272b367258134acfb4e01dd71cfadd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:13.850001 29401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0c7a871f524dccb9453c4dbc604231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:13.850423 29541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0c7a871f524dccb9453c4dbc604231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:13.851116 29681 tablet_service.cc:1467] Processing CreateTablet for tablet 4f0c7a871f524dccb9453c4dbc604231 (DEFAULT_TABLE table=test-table [id=272b367258134acfb4e01dd71cfadd62]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:13.852406 29681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f0c7a871f524dccb9453c4dbc604231. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:13.861656 29771 tablet_bootstrap.cc:492] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Bootstrap starting.
I20260806 20:38:13.864362 29772 tablet_bootstrap.cc:492] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Bootstrap starting.
I20260806 20:38:13.864949 29771 tablet_bootstrap.cc:654] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:13.866567 29773 tablet_bootstrap.cc:492] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Bootstrap starting.
I20260806 20:38:13.866266 29771 log.cc:824] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:13.867769 29772 tablet_bootstrap.cc:654] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:13.869220 29772 log.cc:824] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:13.869597 29773 tablet_bootstrap.cc:654] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:13.870810 29773 log.cc:824] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:13.872385 29772 tablet_bootstrap.cc:492] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: No bootstrap required, opened a new log
I20260806 20:38:13.872633 29772 ts_tablet_manager.cc:1397] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:38:13.872665 29771 tablet_bootstrap.cc:492] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: No bootstrap required, opened a new log
I20260806 20:38:13.872920 29771 ts_tablet_manager.cc:1397] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 20:38:13.873783 29773 tablet_bootstrap.cc:492] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: No bootstrap required, opened a new log
I20260806 20:38:13.874029 29773 ts_tablet_manager.cc:1397] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 20:38:13.880398 29772 raft_consensus.cc:348] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:13.880795 29772 raft_consensus.cc:374] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:13.880894 29772 raft_consensus.cc:729] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90fcf2d4f82f4d978bd2c069b78900d4, State: Initialized, Role: FOLLOWER
I20260806 20:38:13.881466 29772 consensus_queue.cc:260] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:13.882540 29471 heartbeater.cc:503] Master 127.23.200.62:34725 was elected leader, sending a full tablet report...
I20260806 20:38:13.883181 29772 ts_tablet_manager.cc:1428] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Time spent starting tablet: real 0.010s user 0.006s sys 0.006s
I20260806 20:38:13.883428 29771 raft_consensus.cc:348] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:13.883836 29771 raft_consensus.cc:374] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:13.883939 29771 raft_consensus.cc:729] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88f7e0a628a45818fc33325fad2d205, State: Initialized, Role: FOLLOWER
I20260806 20:38:13.884248 29773 raft_consensus.cc:348] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:13.884665 29773 raft_consensus.cc:374] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:13.884547 29771 consensus_queue.cc:260] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:13.884768 29773 raft_consensus.cc:729] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 679457c466aa4793bf2a602771443834, State: Initialized, Role: FOLLOWER
I20260806 20:38:13.885490 29773 consensus_queue.cc:260] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:13.886339 29771 ts_tablet_manager.cc:1428] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260806 20:38:13.886839 29611 heartbeater.cc:503] Master 127.23.200.62:34725 was elected leader, sending a full tablet report...
I20260806 20:38:13.887387 29751 heartbeater.cc:503] Master 127.23.200.62:34725 was elected leader, sending a full tablet report...
I20260806 20:38:13.887923 29773 ts_tablet_manager.cc:1428] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Time spent starting tablet: real 0.014s user 0.013s sys 0.002s
W20260806 20:38:13.923017 29472 tablet.cc:2369] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:38:14.039160 29752 tablet.cc:2369] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:38:14.103116 29614 tablet.cc:2369] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:38:14.151592 29778 raft_consensus.cc:482] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:14.151913 29778 raft_consensus.cc:504] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:14.153757 29778 leader_election.cc:288] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90fcf2d4f82f4d978bd2c069b78900d4 (127.23.200.1:45221), 679457c466aa4793bf2a602771443834 (127.23.200.3:45405)
I20260806 20:38:14.157297 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.154149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:34843 (local address 127.23.200.1:45221)
0806 20:38:14.154420 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:14.154429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:14.155172 (+ 743us) server_negotiation.cc:407] Connection header received
0806 20:38:14.155404 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:14.155408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:14.155460 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:14.155565 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:14.156614 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:14.156620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:14.156624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:14.156650 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:14.156682 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:14.156694 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:14.156822 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:14.156911 (+ 89us) server_negotiation.cc:299] Negotiation successful
0806 20:38:14.156984 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 20:38:14.157382 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.154193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:34871 (local address 127.23.200.3:45405)
0806 20:38:14.154462 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:14.154471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:14.155278 (+ 807us) server_negotiation.cc:407] Connection header received
0806 20:38:14.155581 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:14.155591 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:14.155641 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:14.155745 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:14.156719 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:14.156726 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:14.156735 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:14.156749 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:14.156779 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:14.156789 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:14.156919 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:14.157037 (+ 118us) server_negotiation.cc:299] Negotiation successful
0806 20:38:14.157117 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:38:14.157608 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.154146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:45405 (local address 127.23.200.2:34871)
0806 20:38:14.155153 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.155168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:14.155403 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:14.155779 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:14.155785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:14.155812 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:14.156535 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:14.156537 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:14.156548 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:14.156582 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:14.156898 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:14.156903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:14.156908 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 20:38:14.157073 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 20:38:14.157141 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"thread_start_us":526,"threads_started":1}
I20260806 20:38:14.157691 29782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.153933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.23.200.2:34843)
0806 20:38:14.155060 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.155074 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:14.155220 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:14.155591 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:14.155597 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:14.155620 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:14.156367 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:38:14.156372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:14.156392 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:38:14.156444 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:14.156796 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:14.156800 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:14.156805 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 20:38:14.156994 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 20:38:14.157043 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":781,"threads_started":1}
I20260806 20:38:14.158706 29704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0c7a871f524dccb9453c4dbc604231" candidate_uuid: "a88f7e0a628a45818fc33325fad2d205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679457c466aa4793bf2a602771443834" is_pre_election: true
I20260806 20:38:14.158710 29422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0c7a871f524dccb9453c4dbc604231" candidate_uuid: "a88f7e0a628a45818fc33325fad2d205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" is_pre_election: true
I20260806 20:38:14.159175 29704 raft_consensus.cc:2454] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88f7e0a628a45818fc33325fad2d205 in term 0.
I20260806 20:38:14.159201 29422 raft_consensus.cc:2454] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88f7e0a628a45818fc33325fad2d205 in term 0.
I20260806 20:38:14.159819 29491 leader_election.cc:302] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 679457c466aa4793bf2a602771443834, a88f7e0a628a45818fc33325fad2d205; no voters:
I20260806 20:38:14.160228 29778 raft_consensus.cc:2781] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:14.160377 29778 raft_consensus.cc:482] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:14.160487 29778 raft_consensus.cc:3037] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:14.162747 29778 raft_consensus.cc:504] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:14.163451 29778 leader_election.cc:288] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [CANDIDATE]: Term 1 election: Requested vote from peers 90fcf2d4f82f4d978bd2c069b78900d4 (127.23.200.1:45221), 679457c466aa4793bf2a602771443834 (127.23.200.3:45405)
I20260806 20:38:14.163832 29422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0c7a871f524dccb9453c4dbc604231" candidate_uuid: "a88f7e0a628a45818fc33325fad2d205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90fcf2d4f82f4d978bd2c069b78900d4"
I20260806 20:38:14.163966 29704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f0c7a871f524dccb9453c4dbc604231" candidate_uuid: "a88f7e0a628a45818fc33325fad2d205" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "679457c466aa4793bf2a602771443834"
I20260806 20:38:14.164044 29422 raft_consensus.cc:3037] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:14.164178 29704 raft_consensus.cc:3037] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:14.167380 29704 raft_consensus.cc:2454] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88f7e0a628a45818fc33325fad2d205 in term 1.
I20260806 20:38:14.167375 29422 raft_consensus.cc:2454] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88f7e0a628a45818fc33325fad2d205 in term 1.
I20260806 20:38:14.167783 29491 leader_election.cc:302] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 679457c466aa4793bf2a602771443834, a88f7e0a628a45818fc33325fad2d205; no voters:
I20260806 20:38:14.168128 29778 raft_consensus.cc:2781] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:14.168601 29778 raft_consensus.cc:686] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 1 LEADER]: Becoming Leader. State: Replica: a88f7e0a628a45818fc33325fad2d205, State: Running, Role: LEADER
I20260806 20:38:14.169055 29778 consensus_queue.cc:237] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } }
I20260806 20:38:14.173540 29272 catalog_manager.cc:5495] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 reported cstate change: term changed from 0 to 1, leader changed from <none> to a88f7e0a628a45818fc33325fad2d205 (127.23.200.2). New cstate: current_term: 1 leader_uuid: "a88f7e0a628a45818fc33325fad2d205" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88f7e0a628a45818fc33325fad2d205" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38019 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:14.207789 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.205742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46562 (local address 127.23.200.2:38019)
0806 20:38:14.205958 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:14.205962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:14.205981 (+ 19us) server_negotiation.cc:407] Connection header received
0806 20:38:14.206170 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:14.206174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:14.206250 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:14.206418 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:14.207215 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:14.207219 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:14.207222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:14.207236 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:14.207267 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:14.207278 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:14.207375 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:14.207455 (+ 80us) server_negotiation.cc:299] Negotiation successful
0806 20:38:14.207506 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:38:14.216090 29422 raft_consensus.cc:1261] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 1 FOLLOWER]: Refusing update from remote peer a88f7e0a628a45818fc33325fad2d205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:14.216087 29704 raft_consensus.cc:1261] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 1 FOLLOWER]: Refusing update from remote peer a88f7e0a628a45818fc33325fad2d205: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:14.217007 29778 consensus_queue.cc:1035] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "90fcf2d4f82f4d978bd2c069b78900d4" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:14.217296 29783 consensus_queue.cc:1035] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [LEADER]: Connected to new peer: Peer: permanent_uuid: "679457c466aa4793bf2a602771443834" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 45405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:14.229087 29788 mvcc.cc:205] Tried to move back new op lower bound from 7315655451498614784 to 7315655451320336384. Current Snapshot: MvccSnapshot[applied={T|T < 7315655451498614784}]
I20260806 20:38:14.241014 29272 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:14.241268 29272 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51354:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:14.244851 29272 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2404fb334e604d62a4cf34424dd98c84: Sending DeleteTablet for 3 replicas of tablet 4f0c7a871f524dccb9453c4dbc604231
I20260806 20:38:14.245996 29401 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0c7a871f524dccb9453c4dbc604231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:14 UTC) from {username='slave'} at 127.0.0.1:33854
I20260806 20:38:14.246433 29681 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0c7a871f524dccb9453c4dbc604231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:14 UTC) from {username='slave'} at 127.0.0.1:36584
I20260806 20:38:14.246788 29541 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f0c7a871f524dccb9453c4dbc604231 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:14 UTC) from {username='slave'} at 127.0.0.1:46558
I20260806 20:38:14.247099 29797 tablet_replica.cc:330] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: stopping tablet replica
I20260806 20:38:14.247560 29797 raft_consensus.cc:2229] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:14.248117 29797 raft_consensus.cc:2258] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:14.248605 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29335
I20260806 20:38:14.259013 29797 ts_tablet_manager.cc:1905] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:14.265542 29798 tablet_replica.cc:330] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: stopping tablet replica
I20260806 20:38:14.265991 29798 raft_consensus.cc:2229] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:14.266569 29798 raft_consensus.cc:2258] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:14.268337 29765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.249825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.23.200.1:45221)
0806 20:38:14.250003 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:14.250009 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:14.265149 (+ 15140us) server_negotiation.cc:407] Connection header received
0806 20:38:14.265635 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:14.265640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:14.265741 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:14.265817 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:14.267529 (+ 1712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:14.267537 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:14.267542 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:14.267562 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:14.267599 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:14.267615 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:14.267719 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:14.267971 (+ 252us) server_negotiation.cc:299] Negotiation successful
0806 20:38:14.268049 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 20:38:14.268603 29798 ts_tablet_manager.cc:1905] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:14.269224 29800 tablet_replica.cc:330] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: stopping tablet replica
I20260806 20:38:14.269366 29466 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:14.269743 29800 raft_consensus.cc:2229] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:14.270411 29800 raft_consensus.cc:2258] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:14.274562 29800 ts_tablet_manager.cc:1905] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:14.275564 29798 ts_tablet_manager.cc:1918] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:14.275715 29798 log.cc:1196] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/4f0c7a871f524dccb9453c4dbc604231
I20260806 20:38:14.276213 29798 ts_tablet_manager.cc:1939] T 4f0c7a871f524dccb9453c4dbc604231 P 679457c466aa4793bf2a602771443834: Deleting consensus metadata
I20260806 20:38:14.277959 29256 catalog_manager.cc:4841] TS 679457c466aa4793bf2a602771443834 (127.23.200.3:45405): tablet 4f0c7a871f524dccb9453c4dbc604231 (table test-table [id=272b367258134acfb4e01dd71cfadd62]) successfully deleted
I20260806 20:38:14.283710 29800 ts_tablet_manager.cc:1918] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:14.283869 29800 log.cc:1196] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/4f0c7a871f524dccb9453c4dbc604231
I20260806 20:38:14.284451 29800 ts_tablet_manager.cc:1939] T 4f0c7a871f524dccb9453c4dbc604231 P a88f7e0a628a45818fc33325fad2d205: Deleting consensus metadata
I20260806 20:38:14.286134 29256 catalog_manager.cc:4841] TS a88f7e0a628a45818fc33325fad2d205 (127.23.200.2:38019): tablet 4f0c7a871f524dccb9453c4dbc604231 (table test-table [id=272b367258134acfb4e01dd71cfadd62]) successfully deleted
I20260806 20:38:14.379338 29797 ts_tablet_manager.cc:1918] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:14.379662 29797 log.cc:1196] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/4f0c7a871f524dccb9453c4dbc604231
I20260806 20:38:14.380391 29797 ts_tablet_manager.cc:1939] T 4f0c7a871f524dccb9453c4dbc604231 P 90fcf2d4f82f4d978bd2c069b78900d4: Deleting consensus metadata
I20260806 20:38:14.380649 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29335
I20260806 20:38:14.403086 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29475
W20260806 20:38:14.403210 29258 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:45221 (error 108)
I20260806 20:38:14.407891 29767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.404918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46566 (local address 127.23.200.2:38019)
0806 20:38:14.405150 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:14.405159 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:14.405187 (+ 28us) server_negotiation.cc:407] Connection header received
0806 20:38:14.405318 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:14.405326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:14.405414 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:14.405549 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:14.406820 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:14.406826 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:14.406830 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:14.406849 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:14.406889 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:14.406911 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:14.407056 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:14.407504 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 20:38:14.407574 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 20:38:14.409117 29606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:14.418418 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.407700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33872)
0806 20:38:14.417957 (+ 10257us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.418140 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10064,"thread_start_us":9953,"threads_started":1}
W20260806 20:38:14.418731 29258 catalog_manager.cc:4584] TS 90fcf2d4f82f4d978bd2c069b78900d4 (127.23.200.1:45221): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f0c7a871f524dccb9453c4dbc604231: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
I20260806 20:38:14.472620 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.472162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33878)
0806 20:38:14.472360 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.472479 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 20:38:14.473659 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.473325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33892)
0806 20:38:14.473458 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.473529 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 20:38:14.528268 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29475
I20260806 20:38:14.534113 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.533488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33902)
0806 20:38:14.533780 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.533913 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 20:38:14.535291 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.534811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33912)
0806 20:38:14.535010 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.535104 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 20:38:14.548790 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29616
I20260806 20:38:14.551476 29770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.549518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.23.200.3:45405)
0806 20:38:14.549702 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:14.549707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:14.549724 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:38:14.549888 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:14.549891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:14.549982 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:14.550064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:14.550892 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:14.550896 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:14.550899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:14.550914 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:14.550943 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:14.550954 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:14.551056 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:14.551115 (+ 59us) server_negotiation.cc:299] Negotiation successful
0806 20:38:14.551157 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:38:14.552079 29746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:14.616014 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.615540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33928)
0806 20:38:14.615765 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.615857 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 20:38:14.617097 29804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:14.616681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45221 (local address 127.0.0.1:33932)
0806 20:38:14.616877 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:14.616958 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45221: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 20:38:14.639317 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29616
I20260806 20:38:14.657007 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29241
I20260806 20:38:14.658172 29306 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:14.765347 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29241
2026-08-06T20:38:14Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2843 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 20:38:14.808924 24352 mini_oidc.cc:152] Starting JWKS server
I20260806 20:38:14.811748 24352 webserver.cc:460] Webserver started at https://127.0.0.1:35137/ using document root <none> and password file <none>
I20260806 20:38:14.812083 24352 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 20:38:14.812737 24352 webserver.cc:460] Webserver started at http://127.0.0.1:39693/ using document root <none> and password file <none>
2026-08-06T20:38:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:14Z Disabled control of system clock
I20260806 20:38:14.840034 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44743
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:44743
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35137/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/testchainca.cert with env {}
W20260806 20:38:15.156193 29814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:15.156520 29814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:15.156689 29814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:15.164498 29814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:15.164633 29814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:15.164695 29814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:15.164775 29814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:15.164842 29814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 20:38:15.164932 29814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 20:38:15.165011 29814 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35137/jwks.json
I20260806 20:38:15.176856 29814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:44743
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35137/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:15.178146 29814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:15.179503 29814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:15.191035 29822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:15.191079 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:15.190994 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:15.195823 29814 server_base.cc:1034] running on GCE node
I20260806 20:38:15.196659 29814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:15.198258 29814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:15.199528 29814 hybrid_clock.cc:648] HybridClock initialized: now 1786048695199451 us; error 44 us; skew 500 ppm
I20260806 20:38:15.199883 29814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:15.202437 29814 webserver.cc:460] Webserver started at http://127.23.200.62:36903/ using document root <none> and password file <none>
I20260806 20:38:15.203132 29814 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:15.203259 29814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:15.203595 29814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:15.206563 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "1eb310a72edb42178d7e0d63d800f4a5"
format_stamp: "Formatted at 2026-08-06 20:38:15 on dist-test-slave-tm1g"
server_key: "8db1c4c926a2b21975441894f50bd89b"
server_key_iv: "845f2f8fff893dad7babe98cbac906e5"
server_key_version: "encryptionkey@0"
I20260806 20:38:15.207343 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "1eb310a72edb42178d7e0d63d800f4a5"
format_stamp: "Formatted at 2026-08-06 20:38:15 on dist-test-slave-tm1g"
server_key: "8db1c4c926a2b21975441894f50bd89b"
server_key_iv: "845f2f8fff893dad7babe98cbac906e5"
server_key_version: "encryptionkey@0"
I20260806 20:38:15.212252 29814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:38:15.215690 29828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:15.216950 29814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:38:15.217146 29814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "1eb310a72edb42178d7e0d63d800f4a5"
format_stamp: "Formatted at 2026-08-06 20:38:15 on dist-test-slave-tm1g"
server_key: "8db1c4c926a2b21975441894f50bd89b"
server_key_iv: "845f2f8fff893dad7babe98cbac906e5"
server_key_version: "encryptionkey@0"
I20260806 20:38:15.217356 29814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:15.234969 29814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:15.235729 29814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:15.236038 29814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:15.253654 29814 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:44743
I20260806 20:38:15.253697 29882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:44743 every 8 connection(s)
I20260806 20:38:15.255067 29814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:15.256773 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29814
I20260806 20:38:15.256971 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:15.257814 24352 external_mini_cluster.cc:1428] Setting key a79beee30c8898335f6e32bedf21f2b1
I20260806 20:38:15.259673 29883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:15.270792 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:15.259809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.23.200.62:44743)
0806 20:38:15.260742 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:15.260752 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:15.260790 (+ 38us) server_negotiation.cc:407] Connection header received
0806 20:38:15.262429 (+ 1639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:15.262450 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:15.262876 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:15.263272 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:15.263819 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.265149 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.266520 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.267162 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.267370 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:15.268572 (+ 1202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:15.268581 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:15.268593 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:15.269129 (+ 536us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:15.269202 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:15.269222 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:15.269361 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:15.269573 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 20:38:15.269858 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":331,"threads_started":1}
I20260806 20:38:15.271139 29883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5: Bootstrap starting.
I20260806 20:38:15.274653 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:38:15.274904 29883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:15.275856 29883 log.cc:824] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:15.278661 29883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5: No bootstrap required, opened a new log
I20260806 20:38:15.285918 29883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb310a72edb42178d7e0d63d800f4a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44743 } }
I20260806 20:38:15.286294 29883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:15.286468 29883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eb310a72edb42178d7e0d63d800f4a5, State: Initialized, Role: FOLLOWER
I20260806 20:38:15.287014 29883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb310a72edb42178d7e0d63d800f4a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44743 } }
I20260806 20:38:15.287298 29883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:15.287431 29883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:15.287559 29883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:15.289532 29883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb310a72edb42178d7e0d63d800f4a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44743 } }
I20260806 20:38:15.289969 29883 leader_election.cc:302] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1eb310a72edb42178d7e0d63d800f4a5; no voters:
I20260806 20:38:15.290551 29883 leader_election.cc:288] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:15.290692 29888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:15.291430 29888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [term 1 LEADER]: Becoming Leader. State: Replica: 1eb310a72edb42178d7e0d63d800f4a5, State: Running, Role: LEADER
I20260806 20:38:15.291960 29888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb310a72edb42178d7e0d63d800f4a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44743 } }
I20260806 20:38:15.292641 29883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:15.294786 29890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1eb310a72edb42178d7e0d63d800f4a5. Latest consensus state: current_term: 1 leader_uuid: "1eb310a72edb42178d7e0d63d800f4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb310a72edb42178d7e0d63d800f4a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44743 } } }
I20260806 20:38:15.295372 29890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:15.295765 29889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1eb310a72edb42178d7e0d63d800f4a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1eb310a72edb42178d7e0d63d800f4a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44743 } } }
I20260806 20:38:15.296017 29889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:15.299233 29898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:15.304011 29898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:15.312958 29898 catalog_manager.cc:1347] Generated new cluster ID: 367183b2c3c6437d885f67bbaa6afc57
I20260806 20:38:15.313114 29898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:15.331008 29898 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:15.332468 29898 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:15.337922 29898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1eb310a72edb42178d7e0d63d800f4a5: Generated new TSK 0
I20260806 20:38:15.338745 29898 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 20:38:15.377178 29917 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 20:38:15.377368 29917 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 20:38:15.378377 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:15.357385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.23.200.62:44743)
0806 20:38:15.357657 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:15.357665 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:15.358114 (+ 449us) server_negotiation.cc:407] Connection header received
0806 20:38:15.358458 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:15.358462 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:15.358545 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:15.358659 (+ 114us) server_negotiation.cc:226] Negotiated authn=JWT
0806 20:38:15.359328 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.360034 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.362129 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.362624 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.362716 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:15.363330 (+ 614us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 20:38:15.377824 (+ 14494us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 20:38:15.378089 (+ 265us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 20:38:15.378532 29886 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40486: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 20:38:15.380537 29916 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:15.357306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44743 (local address 127.0.0.1:40486)
0806 20:38:15.358004 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:15.358024 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:15.358211 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:15.358730 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:15.358735 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:15.358750 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0806 20:38:15.359176 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:15.359189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.360183 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.360188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:15.361980 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:15.361997 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.362786 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.362793 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:15.362997 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:15.363077 (+ 80us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 20:38:15.378091 (+ 15014us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 20:38:15.378460 (+ 369us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.200.62:44743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":342,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.200.62:44743: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35137/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 20:38:15.382660 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29814
I20260806 20:38:15.383692 29877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:15.475709 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29814
2026-08-06T20:38:15Z chronyd exiting
I20260806 20:38:15.527312 24352 test_util.cc:175] -----------------------------------------------
I20260806 20:38:15.527446 24352 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786048651485794-24352-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 20:38:15.531164 24352 mini_oidc.cc:152] Starting JWKS server
I20260806 20:38:15.532917 24352 webserver.cc:460] Webserver started at https://127.0.0.1:38039/ using document root <none> and password file <none>
I20260806 20:38:15.533134 24352 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 20:38:15.533702 24352 webserver.cc:460] Webserver started at http://127.0.0.1:44899/ using document root <none> and password file <none>
2026-08-06T20:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:15Z Disabled control of system clock
I20260806 20:38:15.559208 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46219
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:38249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:46219
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38039/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/testchainca.cert with env {}
W20260806 20:38:15.876356 29925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:15.876629 29925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:15.876747 29925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:15.883867 29925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:15.883966 29925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:15.884059 29925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:15.884171 29925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:15.884255 29925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 20:38:15.884348 29925 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 20:38:15.884406 29925 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38039/jwks.json
I20260806 20:38:15.894361 29925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:46219
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38039/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:15.895618 29925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:15.896917 29925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:15.908978 29931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:15.908928 29932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:15.909024 29934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:15.909817 29925 server_base.cc:1034] running on GCE node
I20260806 20:38:15.910682 29925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:15.912065 29925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:15.913290 29925 hybrid_clock.cc:648] HybridClock initialized: now 1786048695913257 us; error 46 us; skew 500 ppm
I20260806 20:38:15.913635 29925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:15.916014 29925 webserver.cc:460] Webserver started at http://127.23.200.62:42081/ using document root <none> and password file <none>
I20260806 20:38:15.916704 29925 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:15.916843 29925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:15.917179 29925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:15.920007 29925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "a7c16abcdc6c46f4bbfc5c367525f605"
format_stamp: "Formatted at 2026-08-06 20:38:15 on dist-test-slave-tm1g"
server_key: "b0c118151ff8d28cf3d24b5139c61cf0"
server_key_iv: "bd48a9ef4f7122f6d6d012c604ea9347"
server_key_version: "encryptionkey@0"
I20260806 20:38:15.920799 29925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "a7c16abcdc6c46f4bbfc5c367525f605"
format_stamp: "Formatted at 2026-08-06 20:38:15 on dist-test-slave-tm1g"
server_key: "b0c118151ff8d28cf3d24b5139c61cf0"
server_key_iv: "bd48a9ef4f7122f6d6d012c604ea9347"
server_key_version: "encryptionkey@0"
I20260806 20:38:15.925731 29925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 20:38:15.929035 29941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:15.930368 29925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:15.930539 29925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "a7c16abcdc6c46f4bbfc5c367525f605"
format_stamp: "Formatted at 2026-08-06 20:38:15 on dist-test-slave-tm1g"
server_key: "b0c118151ff8d28cf3d24b5139c61cf0"
server_key_iv: "bd48a9ef4f7122f6d6d012c604ea9347"
server_key_version: "encryptionkey@0"
I20260806 20:38:15.930752 29925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:15.953528 29925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:15.954452 29925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:15.954771 29925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:15.973269 29925 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:46219
I20260806 20:38:15.973289 29999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:46219 every 8 connection(s)
I20260806 20:38:15.974656 29925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:15.975927 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 29925
I20260806 20:38:15.976188 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:15.977003 24352 external_mini_cluster.cc:1428] Setting key 9aeb323f35d2f8a6d9f8617b13ec36da
I20260806 20:38:15.979399 30000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:15.990680 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:15.979028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.23.200.62:46219)
0806 20:38:15.980030 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:15.980041 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:15.980083 (+ 42us) server_negotiation.cc:407] Connection header received
0806 20:38:15.981658 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:15.981677 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:15.982116 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:15.982522 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:15.983101 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.984441 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.986434 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:15.987068 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:15.987199 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:15.988435 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:15.988442 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:15.988457 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:15.988998 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:15.989079 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:15.989099 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:15.989180 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:15.989425 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 20:38:15.989761 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":412,"threads_started":1}
I20260806 20:38:15.991909 30000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605: Bootstrap starting.
I20260806 20:38:15.994351 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:38:15.995157 30000 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:15.996119 30000 log.cc:824] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:15.999069 30000 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605: No bootstrap required, opened a new log
I20260806 20:38:16.006054 30000 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46219 } }
I20260806 20:38:16.006420 30000 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:16.006524 30000 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7c16abcdc6c46f4bbfc5c367525f605, State: Initialized, Role: FOLLOWER
I20260806 20:38:16.007092 30000 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46219 } }
I20260806 20:38:16.007325 30000 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:16.007464 30000 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:16.007633 30000 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:16.009945 30000 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46219 } }
I20260806 20:38:16.010484 30000 leader_election.cc:302] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7c16abcdc6c46f4bbfc5c367525f605; no voters:
I20260806 20:38:16.011063 30000 leader_election.cc:288] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:16.011193 30005 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:16.011904 30005 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [term 1 LEADER]: Becoming Leader. State: Replica: a7c16abcdc6c46f4bbfc5c367525f605, State: Running, Role: LEADER
I20260806 20:38:16.012568 30005 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46219 } }
I20260806 20:38:16.013239 30000 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:16.015419 30007 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7c16abcdc6c46f4bbfc5c367525f605. Latest consensus state: current_term: 1 leader_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46219 } } }
I20260806 20:38:16.015854 30007 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:16.016276 30006 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7c16abcdc6c46f4bbfc5c367525f605" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46219 } } }
I20260806 20:38:16.016708 30006 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:16.018098 30015 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:16.023358 30015 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:16.032207 30015 catalog_manager.cc:1347] Generated new cluster ID: d21b0f3cad584a72a8134bc1b903e020
I20260806 20:38:16.032317 30015 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:16.044534 30015 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:16.045742 30015 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:16.053231 30015 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7c16abcdc6c46f4bbfc5c367525f605: Generated new TSK 0
I20260806 20:38:16.053997 30015 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 20:38:16.088819 30031 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 20:38:16.088959 30031 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260806 20:38:16.090160 30030 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:16.070193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46219 (local address 127.0.0.1:41534)
0806 20:38:16.070875 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:16.070890 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:16.071018 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:16.071471 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:16.071474 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:16.071484 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0806 20:38:16.071710 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:16.071718 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.073143 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.073146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:16.074232 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:16.074242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.075294 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.075299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:16.075441 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:16.075472 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 20:38:16.089683 (+ 14211us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 20:38:16.089875 (+ 192us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.200.62:46219: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":401,"threads_started":1}
I20260806 20:38:16.090166 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:16.070208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41534 (local address 127.23.200.62:46219)
0806 20:38:16.070404 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:16.070411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:16.070991 (+ 580us) server_negotiation.cc:407] Connection header received
0806 20:38:16.071206 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:16.071212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:16.071301 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:16.071457 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0806 20:38:16.071867 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.073007 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.074408 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.075115 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.075256 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:16.075735 (+ 479us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 20:38:16.089472 (+ 13737us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 20:38:16.089854 (+ 382us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41534: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 20:38:16.090291 30003 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41534: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38039/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260806 20:38:16.091349 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 29925
I20260806 20:38:16.092588 29994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:16.184450 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 29925
2026-08-06T20:38:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (706 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 20:38:16.237706 24352 mini_oidc.cc:152] Starting JWKS server
I20260806 20:38:16.239686 24352 webserver.cc:460] Webserver started at https://127.0.0.1:45695/ using document root <none> and password file <none>
I20260806 20:38:16.239939 24352 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 20:38:16.240567 24352 webserver.cc:460] Webserver started at http://127.0.0.1:44149/ using document root <none> and password file <none>
2026-08-06T20:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:16Z Disabled control of system clock
I20260806 20:38:16.267966 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44603
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:33185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:44603
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45695/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 20:38:16.586527 30038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:16.586862 30038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:16.587026 30038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:16.594064 30038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:16.594202 30038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:16.594292 30038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:16.594411 30038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:16.594491 30038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 20:38:16.594579 30038 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 20:38:16.594659 30038 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45695/jwks.json
I20260806 20:38:16.603827 30038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:44603
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45695/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:16.605134 30038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:16.607026 30038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:16.619032 30044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:16.619122 30043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:16.619143 30046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:16.620251 30038 server_base.cc:1034] running on GCE node
I20260806 20:38:16.621011 30038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:16.622382 30038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:16.623592 30038 hybrid_clock.cc:648] HybridClock initialized: now 1786048696623539 us; error 48 us; skew 500 ppm
I20260806 20:38:16.623935 30038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:16.626127 30038 webserver.cc:460] Webserver started at http://127.23.200.62:36973/ using document root <none> and password file <none>
I20260806 20:38:16.626873 30038 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:16.626991 30038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:16.627336 30038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:16.630162 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7"
format_stamp: "Formatted at 2026-08-06 20:38:16 on dist-test-slave-tm1g"
server_key: "f2e35f63ea3c76598d6f4805f77774f0"
server_key_iv: "97907a3a8e3421f305de012509ff6e66"
server_key_version: "encryptionkey@0"
I20260806 20:38:16.631043 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7"
format_stamp: "Formatted at 2026-08-06 20:38:16 on dist-test-slave-tm1g"
server_key: "f2e35f63ea3c76598d6f4805f77774f0"
server_key_iv: "97907a3a8e3421f305de012509ff6e66"
server_key_version: "encryptionkey@0"
I20260806 20:38:16.636068 30038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:38:16.639412 30052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:16.640856 30038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:38:16.641052 30038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7"
format_stamp: "Formatted at 2026-08-06 20:38:16 on dist-test-slave-tm1g"
server_key: "f2e35f63ea3c76598d6f4805f77774f0"
server_key_iv: "97907a3a8e3421f305de012509ff6e66"
server_key_version: "encryptionkey@0"
I20260806 20:38:16.641266 30038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:16.655759 30038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:16.656528 30038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:16.656831 30038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:16.675628 30038 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:44603
I20260806 20:38:16.675647 30107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:44603 every 8 connection(s)
I20260806 20:38:16.677011 30038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:16.680691 30108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:16.685745 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30038
I20260806 20:38:16.685962 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:16.686817 24352 external_mini_cluster.cc:1428] Setting key d8c97549c0165c73a745622fdd5d5eda
I20260806 20:38:16.692118 30108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7: Bootstrap starting.
I20260806 20:38:16.695248 30108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:16.696287 30108 log.cc:824] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:16.699231 30108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7: No bootstrap required, opened a new log
I20260806 20:38:16.700090 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:16.688714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53282 (local address 127.23.200.62:44603)
0806 20:38:16.689749 (+ 1035us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:16.689759 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:16.689820 (+ 61us) server_negotiation.cc:407] Connection header received
0806 20:38:16.691552 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:16.691573 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:16.691991 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:16.692356 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:16.692944 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.694229 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.695655 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.696295 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.696529 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:16.697661 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:16.697669 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:16.697681 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:16.698177 (+ 496us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:16.698255 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:16.698275 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:16.698457 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:16.698689 (+ 232us) server_negotiation.cc:299] Negotiation successful
0806 20:38:16.698985 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":407,"threads_started":1}
I20260806 20:38:16.703843 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:38:16.707707 30108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44603 } }
I20260806 20:38:16.708065 30108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:16.708179 30108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc6a986b24b46ea93b6f4b26ad34fe7, State: Initialized, Role: FOLLOWER
I20260806 20:38:16.708704 30108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44603 } }
I20260806 20:38:16.708948 30108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:16.709096 30108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:16.709268 30108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:16.711606 30108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44603 } }
I20260806 20:38:16.712134 30108 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc6a986b24b46ea93b6f4b26ad34fe7; no voters:
I20260806 20:38:16.712714 30108 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:16.712863 30113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:16.713572 30113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [term 1 LEADER]: Becoming Leader. State: Replica: 9cc6a986b24b46ea93b6f4b26ad34fe7, State: Running, Role: LEADER
I20260806 20:38:16.714104 30113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44603 } }
I20260806 20:38:16.714720 30108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:16.717324 30114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44603 } } }
I20260806 20:38:16.717303 30115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc6a986b24b46ea93b6f4b26ad34fe7. Latest consensus state: current_term: 1 leader_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc6a986b24b46ea93b6f4b26ad34fe7" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44603 } } }
I20260806 20:38:16.717970 30114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:16.717991 30115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:16.719103 30119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:16.724227 30119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:16.733031 30119 catalog_manager.cc:1347] Generated new cluster ID: bc3c064665964b0bbf5a605d81a34664
I20260806 20:38:16.733160 30119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:16.782819 30119 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:16.784123 30119 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:16.795926 30119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc6a986b24b46ea93b6f4b26ad34fe7: Generated new TSK 0
I20260806 20:38:16.796757 30119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:38:16.830497 30111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:16.822138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53290 (local address 127.23.200.62:44603)
0806 20:38:16.822422 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:16.822428 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:16.822761 (+ 333us) server_negotiation.cc:407] Connection header received
0806 20:38:16.822998 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:16.823002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:16.823069 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:16.823208 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0806 20:38:16.823768 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.824721 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.826788 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.827234 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.827328 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:16.827923 (+ 595us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 20:38:16.829786 (+ 1863us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 20:38:16.830127 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 20:38:16.830677 30137 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:16.822020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44603 (local address 127.0.0.1:53290)
0806 20:38:16.822645 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:16.822673 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:16.822821 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:16.823245 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:16.823250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:16.823265 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0806 20:38:16.823602 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:16.823615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.824880 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.824886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:16.826633 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:16.826651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:16.827423 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:16.827430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:16.827628 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:16.827673 (+ 45us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 20:38:16.829985 (+ 2312us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 20:38:16.830218 (+ 233us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.200.62:44603: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":343,"threads_started":1}
W20260806 20:38:16.830996 30111 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53290: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45695/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260806 20:38:16.831727 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30038
I20260806 20:38:16.833597 30103 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 20:38:16.843513 30138 webserver.cc:533] Webserver: sslize failed:
E20260806 20:38:16.843602 30138 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 20:38:16.929991 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30038
2026-08-06T20:38:16Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (748 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T20:38:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:16Z Disabled control of system clock
I20260806 20:38:17.013501 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:34909
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:38141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:34909
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/testchainca.cert with env {}
W20260806 20:38:17.338053 30143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:17.338388 30143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:17.338485 30143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:17.345577 30143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:17.345669 30143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:17.345705 30143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:17.345772 30143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:17.345815 30143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 20:38:17.345876 30143 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 20:38:17.345916 30143 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 20:38:17.355557 30143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:34909
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:34909
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:17.356760 30143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:17.357964 30143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:17.369866 30151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:17.369815 30148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:17.369828 30149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:17.370751 30143 server_base.cc:1034] running on GCE node
I20260806 20:38:17.371529 30143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:17.372884 30143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:17.374106 30143 hybrid_clock.cc:648] HybridClock initialized: now 1786048697374063 us; error 40 us; skew 500 ppm
I20260806 20:38:17.374504 30143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:38:17.376678 30143 webserver.cc:460] Webserver started at http://127.23.200.62:45467/ using document root <none> and password file <none>
I20260806 20:38:17.377343 30143 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:17.377496 30143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:17.377802 30143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:17.380997 30143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "f66cd1436e2c4175822e3f3cd8574419"
format_stamp: "Formatted at 2026-08-06 20:38:17 on dist-test-slave-tm1g"
server_key: "dbae464a921799e73d477f6ee2505e70"
server_key_iv: "517f89072270f68a1fecd2f2f9d961f2"
server_key_version: "encryptionkey@0"
I20260806 20:38:17.381925 30143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "f66cd1436e2c4175822e3f3cd8574419"
format_stamp: "Formatted at 2026-08-06 20:38:17 on dist-test-slave-tm1g"
server_key: "dbae464a921799e73d477f6ee2505e70"
server_key_iv: "517f89072270f68a1fecd2f2f9d961f2"
server_key_version: "encryptionkey@0"
I20260806 20:38:17.386916 30143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:38:17.390125 30157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:17.391489 30143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:17.391690 30143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "f66cd1436e2c4175822e3f3cd8574419"
format_stamp: "Formatted at 2026-08-06 20:38:17 on dist-test-slave-tm1g"
server_key: "dbae464a921799e73d477f6ee2505e70"
server_key_iv: "517f89072270f68a1fecd2f2f9d961f2"
server_key_version: "encryptionkey@0"
I20260806 20:38:17.391889 30143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:17.408095 30143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:17.408838 30143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:17.409119 30143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:17.427708 30143 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:34909
I20260806 20:38:17.427726 30210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:34909 every 8 connection(s)
I20260806 20:38:17.429039 30143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:17.431039 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30143
I20260806 20:38:17.431257 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:17.432104 24352 external_mini_cluster.cc:1428] Setting key f1846c60b83db3cd176d5544c87a745a
I20260806 20:38:17.433462 30211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:17.445487 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:17.434126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.23.200.62:34909)
0806 20:38:17.435146 (+ 1020us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:17.435156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:17.435195 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:38:17.436669 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:17.436690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:17.437131 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:17.437440 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:17.438047 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:17.439406 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:17.441235 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:17.441860 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:17.441996 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:17.443233 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:17.443243 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:17.443256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:38:17.443804 (+ 548us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:17.443891 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:38:17.443911 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:17.444010 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:17.444247 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 20:38:17.444587 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":419,"threads_started":1}
I20260806 20:38:17.446923 30211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419: Bootstrap starting.
I20260806 20:38:17.449255 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:38:17.450133 30211 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:17.451141 30211 log.cc:824] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:17.453989 30211 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419: No bootstrap required, opened a new log
I20260806 20:38:17.461154 30211 raft_consensus.cc:348] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66cd1436e2c4175822e3f3cd8574419" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34909 } }
I20260806 20:38:17.461567 30211 raft_consensus.cc:374] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:17.461669 30211 raft_consensus.cc:729] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f66cd1436e2c4175822e3f3cd8574419, State: Initialized, Role: FOLLOWER
I20260806 20:38:17.462229 30211 consensus_queue.cc:260] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66cd1436e2c4175822e3f3cd8574419" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34909 } }
I20260806 20:38:17.462520 30211 raft_consensus.cc:388] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:17.462670 30211 raft_consensus.cc:482] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:17.462863 30211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:17.464885 30211 raft_consensus.cc:504] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66cd1436e2c4175822e3f3cd8574419" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34909 } }
I20260806 20:38:17.465378 30211 leader_election.cc:302] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f66cd1436e2c4175822e3f3cd8574419; no voters:
I20260806 20:38:17.465951 30211 leader_election.cc:288] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:17.466117 30216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:17.466856 30216 raft_consensus.cc:686] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [term 1 LEADER]: Becoming Leader. State: Replica: f66cd1436e2c4175822e3f3cd8574419, State: Running, Role: LEADER
I20260806 20:38:17.467494 30216 consensus_queue.cc:237] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66cd1436e2c4175822e3f3cd8574419" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34909 } }
I20260806 20:38:17.468127 30211 sys_catalog.cc:564] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:17.470288 30218 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f66cd1436e2c4175822e3f3cd8574419. Latest consensus state: current_term: 1 leader_uuid: "f66cd1436e2c4175822e3f3cd8574419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66cd1436e2c4175822e3f3cd8574419" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34909 } } }
I20260806 20:38:17.470683 30218 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:17.471073 30217 sys_catalog.cc:455] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f66cd1436e2c4175822e3f3cd8574419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f66cd1436e2c4175822e3f3cd8574419" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 34909 } } }
I20260806 20:38:17.471447 30217 sys_catalog.cc:458] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:17.472415 30225 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:17.476738 30225 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:17.485152 30225 catalog_manager.cc:1347] Generated new cluster ID: f4b592ed99244aec8236d1a3499eb0f6
I20260806 20:38:17.485267 30225 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:17.493219 30225 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:17.494464 30225 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:17.502400 30225 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f66cd1436e2c4175822e3f3cd8574419: Generated new TSK 0
I20260806 20:38:17.503142 30225 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:38:18.769992 30214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:17.517733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45408 (local address 127.23.200.62:34909)
0806 20:38:17.517955 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:17.517964 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:17.518468 (+ 504us) server_negotiation.cc:407] Connection header received
0806 20:38:17.518679 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:17.518682 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:17.518752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:17.518880 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0806 20:38:17.519387 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:17.520121 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:17.522174 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:17.522677 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:17.522774 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:17.523330 (+ 556us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 20:38:18.769403 (+1246073us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 20:38:18.769745 (+ 342us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45408: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 20:38:18.770143 30214 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45408: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 20:38:18.770361 30241 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:17.517719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:34909 (local address 127.0.0.1:45408)
0806 20:38:17.518365 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:17.518385 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:17.518530 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:17.518932 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:17.518937 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:17.518950 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 20:38:17.519240 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:17.519252 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:17.520274 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:17.520280 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:17.522034 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:17.522049 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:17.522852 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:17.522859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:17.523051 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:17.523109 (+ 58us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 20:38:18.769713 (+1246604us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 20:38:18.769964 (+ 251us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.200.62:34909: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":315,"threads_started":1}
I20260806 20:38:18.771876 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30143
I20260806 20:38:18.772986 30206 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:18.875452 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30143
2026-08-06T20:38:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1927 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (409 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (423 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:19 dist-test-slave-tm1g krb5kdc[30257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:19 dist-test-slave-tm1g krb5kdc[30257](info): set up 2 sockets
Aug 06 20:38:19 dist-test-slave-tm1g krb5kdc[30257](info): commencing operation
krb5kdc: starting...
W20260806 20:38:20.814404 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:20 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:38:21.010850 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:38049
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:21.325284 30272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:21.325593 30272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:21.325774 30272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:21.332939 30272 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:21.333042 30272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:21.333155 30272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:21.333259 30272 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:21.333338 30272 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:21.343600 30272 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:38049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:21.344970 30272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:21.346357 30272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:21.357962 30278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:21.358038 30279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:21.358072 30281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:21.359129 30272 server_base.cc:1034] running on GCE node
I20260806 20:38:21.359870 30272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:21.361202 30272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:21.362413 30272 hybrid_clock.cc:648] HybridClock initialized: now 1786048701362356 us; error 55 us; skew 500 ppm
I20260806 20:38:21.362751 30272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:21 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048701, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:21.365736 30272 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:38:21.367228 30272 webserver.cc:460] Webserver started at http://127.23.200.62:36001/ using document root <none> and password file <none>
I20260806 20:38:21.367866 30272 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:21.367976 30272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:21.368237 30272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:21.371183 30272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "04fbca227cc74bbebd132c1e0f8688d2"
format_stamp: "Formatted at 2026-08-06 20:38:21 on dist-test-slave-tm1g"
server_key: "d43186f8c50a67d8d9b1e8f9d12b7b46"
server_key_iv: "dc2e42e9eb677be6680c364d8571e999"
server_key_version: "encryptionkey@0"
I20260806 20:38:21.371876 30272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "04fbca227cc74bbebd132c1e0f8688d2"
format_stamp: "Formatted at 2026-08-06 20:38:21 on dist-test-slave-tm1g"
server_key: "d43186f8c50a67d8d9b1e8f9d12b7b46"
server_key_iv: "dc2e42e9eb677be6680c364d8571e999"
server_key_version: "encryptionkey@0"
I20260806 20:38:21.376875 30272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:38:21.380106 30288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:21.381433 30272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:38:21.381615 30272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "04fbca227cc74bbebd132c1e0f8688d2"
format_stamp: "Formatted at 2026-08-06 20:38:21 on dist-test-slave-tm1g"
server_key: "d43186f8c50a67d8d9b1e8f9d12b7b46"
server_key_iv: "dc2e42e9eb677be6680c364d8571e999"
server_key_version: "encryptionkey@0"
I20260806 20:38:21.381809 30272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:21.393472 30272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:21.397293 30272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:21.397581 30272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:21.416164 30272 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:38049
I20260806 20:38:21.416199 30341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:38049 every 8 connection(s)
I20260806 20:38:21.417559 30272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:21.421303 30342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:21.427963 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30272
I20260806 20:38:21.428228 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:21.429032 24352 external_mini_cluster.cc:1428] Setting key fe1bacd2ef204df2f39bc2d3fb01516c
I20260806 20:38:21.432616 30342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2: Bootstrap starting.
I20260806 20:38:21.436199 30342 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:21.437287 30342 log.cc:824] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:21.440047 30342 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2: No bootstrap required, opened a new log
Aug 06 20:38:21 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:21.448405 30342 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fbca227cc74bbebd132c1e0f8688d2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38049 } }
I20260806 20:38:21.448802 30342 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:21.448921 30342 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04fbca227cc74bbebd132c1e0f8688d2, State: Initialized, Role: FOLLOWER
I20260806 20:38:21.449581 30342 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fbca227cc74bbebd132c1e0f8688d2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38049 } }
I20260806 20:38:21.449859 30342 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:21.450026 30342 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:21.450245 30342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:21.451469 30345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:21.431156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.23.200.62:38049)
0806 20:38:21.431918 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:21.431930 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:21.431973 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:38:21.433725 (+ 1752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:21.433747 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:21.434136 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:21.434520 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:21.436257 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:21.437646 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:21.439003 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:21.439657 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:21.439845 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:21.443804 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:21.443812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:21.443825 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:21.443865 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:21.447129 (+ 3264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:21.447820 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:21.447828 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:21.447834 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:21.448027 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:21.448439 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:21.448443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:21.448445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:21.449092 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:21.449462 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:21.449892 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 20:38:21.450404 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":377,"threads_started":1}
I20260806 20:38:21.452744 30342 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fbca227cc74bbebd132c1e0f8688d2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38049 } }
I20260806 20:38:21.453266 30342 leader_election.cc:302] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04fbca227cc74bbebd132c1e0f8688d2; no voters:
I20260806 20:38:21.453990 30342 leader_election.cc:288] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:21.454124 30347 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:21.454916 30347 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [term 1 LEADER]: Becoming Leader. State: Replica: 04fbca227cc74bbebd132c1e0f8688d2, State: Running, Role: LEADER
I20260806 20:38:21.455498 30347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fbca227cc74bbebd132c1e0f8688d2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38049 } }
I20260806 20:38:21.456283 30342 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:21.458424 30350 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04fbca227cc74bbebd132c1e0f8688d2. Latest consensus state: current_term: 1 leader_uuid: "04fbca227cc74bbebd132c1e0f8688d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fbca227cc74bbebd132c1e0f8688d2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38049 } } }
I20260806 20:38:21.458868 30350 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:21.459643 30348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04fbca227cc74bbebd132c1e0f8688d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04fbca227cc74bbebd132c1e0f8688d2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 38049 } } }
I20260806 20:38:21.460464 30348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:21.460559 30356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:21.466702 30356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:21.475464 30356 catalog_manager.cc:1347] Generated new cluster ID: b995056a1ba240348f4b161751831462
I20260806 20:38:21.475576 30356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:21.504559 30356 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:21.505972 30356 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:21.511696 30356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04fbca227cc74bbebd132c1e0f8688d2: Generated new TSK 0
I20260806 20:38:21.512485 30356 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:38:21.571987 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38049
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:21.908300 30370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:21.908545 30370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:21.908633 30370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:21.915771 30370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:21.915875 30370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:21.915983 30370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:21.925153 30370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:21.926275 30370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:21.927521 30370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:21.940232 30379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:21.940279 30376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:21.940315 30375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:21.941192 30370 server_base.cc:1034] running on GCE node
I20260806 20:38:21.941741 30370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:21.942612 30370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:21.943888 30370 hybrid_clock.cc:648] HybridClock initialized: now 1786048701943851 us; error 42 us; skew 500 ppm
I20260806 20:38:21.944144 30370 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:21 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048701, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:21.947106 30370 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:38:21.948446 30370 webserver.cc:460] Webserver started at http://127.23.200.1:41065/ using document root <none> and password file <none>
I20260806 20:38:21.949059 30370 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:21.949155 30370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:21.949405 30370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:21.952217 30370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "5b0bbb93ac974e769e23f84961bef733"
format_stamp: "Formatted at 2026-08-06 20:38:21 on dist-test-slave-tm1g"
server_key: "c4b50cecce8e402844debfacfb2ff62c"
server_key_iv: "697e7000c5cd66063109b6d2abdc0fc8"
server_key_version: "encryptionkey@0"
I20260806 20:38:21.953006 30370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "5b0bbb93ac974e769e23f84961bef733"
format_stamp: "Formatted at 2026-08-06 20:38:21 on dist-test-slave-tm1g"
server_key: "c4b50cecce8e402844debfacfb2ff62c"
server_key_iv: "697e7000c5cd66063109b6d2abdc0fc8"
server_key_version: "encryptionkey@0"
I20260806 20:38:21.958376 30370 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:38:21.961701 30386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:21.963063 30370 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 20:38:21.963232 30370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "5b0bbb93ac974e769e23f84961bef733"
format_stamp: "Formatted at 2026-08-06 20:38:21 on dist-test-slave-tm1g"
server_key: "c4b50cecce8e402844debfacfb2ff62c"
server_key_iv: "697e7000c5cd66063109b6d2abdc0fc8"
server_key_version: "encryptionkey@0"
I20260806 20:38:21.963447 30370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:21.983057 30370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:21.987187 30370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:21.987497 30370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:21.988435 30370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:21.989722 30370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:21.989827 30370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:21.989984 30370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:21.990057 30370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:22.023082 30370 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:38401
I20260806 20:38:22.023128 30504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:38401 every 8 connection(s)
I20260806 20:38:22.024343 30370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:22.026791 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30370
I20260806 20:38:22.027038 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:22.027671 24352 external_mini_cluster.cc:1428] Setting key ee9f26c6e4a46a026ef49586d105dc06
Aug 06 20:38:22 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048701, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:22.046469 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:22.026943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49771 (local address 127.23.200.62:38049)
0806 20:38:22.028140 (+ 1197us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:22.028147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:22.028912 (+ 765us) server_negotiation.cc:407] Connection header received
0806 20:38:22.031369 (+ 2457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:22.031373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:22.031438 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:22.031530 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:22.033563 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.034291 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.036810 (+ 2519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.037215 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.037360 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:22.040837 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:22.040842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:22.040845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:22.040876 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:22.043373 (+ 2497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:22.044095 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:22.044101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:22.044104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:22.044239 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:22.044684 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:22.044689 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:22.044692 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:22.044966 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:22.045127 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:22.045813 (+ 686us) server_negotiation.cc:299] Negotiation successful
0806 20:38:22.046063 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1059,"thread_start_us":959,"threads_started":1}
I20260806 20:38:22.047405 30508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:22.027541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38049 (local address 127.23.200.1:49771)
0806 20:38:22.028742 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:22.028801 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:22.031074 (+ 2273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:22.031801 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:22.031815 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:22.032595 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:22.033371 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:22.033398 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.034528 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.034536 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:22.036668 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:22.036685 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.037470 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.037478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:22.037709 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:22.038458 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:22.038484 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:22.040614 (+ 2130us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:22.043526 (+ 2912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:22.043535 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:22.043550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:22.043959 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:22.044353 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:22.044358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:22.044361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:22.044560 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:22.045098 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:22.045105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:22.045455 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 20:38:22.045728 (+ 273us) client_negotiation.cc:240] Negotiation successful
0806 20:38:22.046245 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":901,"thread_start_us":650,"threads_started":1}
I20260806 20:38:22.049064 30505 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38049
I20260806 20:38:22.049396 30505 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:38:22.050271 30505 heartbeater.cc:511] Master 127.23.200.62:38049 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
I20260806 20:38:22.052511 30306 ts_manager.cc:194] Registered new tserver with Master: 5b0bbb93ac974e769e23f84961bef733 (127.23.200.1:38401)
I20260806 20:38:22.053920 30306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:49771
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:38:22.076992 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38049
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:22.395752 30511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:22.396015 30511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:22.396176 30511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:22.403249 30511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:22.403344 30511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:22.403558 30511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:22.412993 30511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:22.414186 30511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:22.415565 30511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:22.430225 30517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:22.430248 30516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:22.430981 30519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:22.431380 30511 server_base.cc:1034] running on GCE node
I20260806 20:38:22.431998 30511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:22.432901 30511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:22.434149 30511 hybrid_clock.cc:648] HybridClock initialized: now 1786048702434101 us; error 44 us; skew 500 ppm
I20260806 20:38:22.434506 30511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:22 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048702, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:22.437485 30511 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:38:22.438874 30511 webserver.cc:460] Webserver started at http://127.23.200.2:41019/ using document root <none> and password file <none>
I20260806 20:38:22.439481 30511 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:22.439589 30511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:22.439836 30511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:22.442616 30511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "2f0b9e2c25264ff0a766af2abf930dee"
format_stamp: "Formatted at 2026-08-06 20:38:22 on dist-test-slave-tm1g"
server_key: "af7a76cdde7d365d3bc2cbf67e500ef5"
server_key_iv: "82c30159e656be867f27dcf3a79bc526"
server_key_version: "encryptionkey@0"
I20260806 20:38:22.443318 30511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "2f0b9e2c25264ff0a766af2abf930dee"
format_stamp: "Formatted at 2026-08-06 20:38:22 on dist-test-slave-tm1g"
server_key: "af7a76cdde7d365d3bc2cbf67e500ef5"
server_key_iv: "82c30159e656be867f27dcf3a79bc526"
server_key_version: "encryptionkey@0"
I20260806 20:38:22.448325 30511 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:38:22.451725 30526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:22.452971 30511 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:38:22.453145 30511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "2f0b9e2c25264ff0a766af2abf930dee"
format_stamp: "Formatted at 2026-08-06 20:38:22 on dist-test-slave-tm1g"
server_key: "af7a76cdde7d365d3bc2cbf67e500ef5"
server_key_iv: "82c30159e656be867f27dcf3a79bc526"
server_key_version: "encryptionkey@0"
I20260806 20:38:22.453339 30511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:22.475301 30511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:22.479203 30511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:22.479491 30511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:22.480417 30511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:22.481701 30511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:22.481776 30511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:22.481942 30511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:22.481993 30511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:22.514602 30511 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:43719
I20260806 20:38:22.514699 30645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:43719 every 8 connection(s)
I20260806 20:38:22.515904 30511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:22.525113 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30511
I20260806 20:38:22.525426 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:22.525873 24352 external_mini_cluster.cc:1428] Setting key 85505ce7f4571c7711e8e1dc547a24df
Aug 06 20:38:22 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048702, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:22.537487 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:22.518464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:58335 (local address 127.23.200.62:38049)
0806 20:38:22.518666 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:22.518671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:22.519835 (+ 1164us) server_negotiation.cc:407] Connection header received
0806 20:38:22.521597 (+ 1762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:22.521602 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:22.521670 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:22.521758 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:22.523481 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.524214 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.525820 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.526395 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.526547 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:22.530535 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:22.530543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:22.530551 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:22.530595 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:22.534156 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:22.535071 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:22.535079 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:22.535084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:22.535223 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:22.535787 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:22.535795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:22.535799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:22.536185 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:22.536356 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:22.537074 (+ 718us) server_negotiation.cc:299] Negotiation successful
0806 20:38:22.537241 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 20:38:22.538975 30649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:22.518944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38049 (local address 127.23.200.2:58335)
0806 20:38:22.519677 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:22.519713 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:22.521351 (+ 1638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:22.521966 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:22.521973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:22.522731 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:22.523292 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:22.523320 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.524349 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.524353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:22.525677 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:22.525689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:22.526614 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:22.526619 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:22.526918 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:22.527736 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:22.527758 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:22.530265 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:22.534319 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:22.534330 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:22.534346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:22.534934 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:22.535384 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:22.535390 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:22.535392 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:22.535638 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:22.536317 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:22.536322 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:22.536665 (+ 343us) client_negotiation.cc:769] Sending connection context
0806 20:38:22.536970 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 20:38:22.537538 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":303,"threads_started":1}
I20260806 20:38:22.540414 30646 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38049
I20260806 20:38:22.540709 30646 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:22.541388 30646 heartbeater.cc:511] Master 127.23.200.62:38049 requested a full tablet report, sending...
I20260806 20:38:22.542835 30306 ts_manager.cc:194] Registered new tserver with Master: 2f0b9e2c25264ff0a766af2abf930dee (127.23.200.2:43719)
I20260806 20:38:22.543641 30306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:58335
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:38:22.573236 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38049
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:40875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:22.891911 30652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:22.892174 30652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:22.892267 30652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:22.899261 30652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:22.899367 30652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:22.899477 30652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:22.908748 30652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:38049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:22.909932 30652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:22.911242 30652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:22.924119 30657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:22.924181 30658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:22.924157 30660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:22.925024 30652 server_base.cc:1034] running on GCE node
I20260806 20:38:22.925688 30652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:22.926651 30652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:22.927887 30652 hybrid_clock.cc:648] HybridClock initialized: now 1786048702927846 us; error 48 us; skew 500 ppm
I20260806 20:38:22.928195 30652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:22 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048702, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:22.931244 30652 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:38:22.932663 30652 webserver.cc:460] Webserver started at http://127.23.200.3:45461/ using document root <none> and password file <none>
I20260806 20:38:22.933362 30652 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:22.933507 30652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:22.933820 30652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:22.936892 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "61a82aebe3d445928b86a28185c856ab"
format_stamp: "Formatted at 2026-08-06 20:38:22 on dist-test-slave-tm1g"
server_key: "ffcd2d4a498769d0a0b56aa0c9cd3722"
server_key_iv: "880bd93fb2010251bb3bdc8a83a15a37"
server_key_version: "encryptionkey@0"
I20260806 20:38:22.937686 30652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "61a82aebe3d445928b86a28185c856ab"
format_stamp: "Formatted at 2026-08-06 20:38:22 on dist-test-slave-tm1g"
server_key: "ffcd2d4a498769d0a0b56aa0c9cd3722"
server_key_iv: "880bd93fb2010251bb3bdc8a83a15a37"
server_key_version: "encryptionkey@0"
I20260806 20:38:22.943101 30652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260806 20:38:22.946394 30668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:22.947876 30652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:38:22.948033 30652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "61a82aebe3d445928b86a28185c856ab"
format_stamp: "Formatted at 2026-08-06 20:38:22 on dist-test-slave-tm1g"
server_key: "ffcd2d4a498769d0a0b56aa0c9cd3722"
server_key_iv: "880bd93fb2010251bb3bdc8a83a15a37"
server_key_version: "encryptionkey@0"
I20260806 20:38:22.948252 30652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:22.962827 30652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:22.966594 30652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:22.966874 30652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:22.967867 30652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:22.969197 30652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:22.969300 30652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:22.969444 30652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:22.969525 30652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:23.002663 30652 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:41919
I20260806 20:38:23.002708 30785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:41919 every 8 connection(s)
I20260806 20:38:23.003958 30652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:23.011055 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30652
I20260806 20:38:23.011242 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:23.011673 24352 external_mini_cluster.cc:1428] Setting key d5e7076063ad43fa8a9f408ae3e71d08
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048702, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:23.023566 30507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.006490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:48831 (local address 127.23.200.62:38049)
0806 20:38:23.006665 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.006669 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.007780 (+ 1111us) server_negotiation.cc:407] Connection header received
0806 20:38:23.009628 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.009636 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.009720 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.009838 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.011490 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.012347 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.014256 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.014891 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.015036 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.018228 (+ 3192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.018233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.018236 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.018266 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.020727 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.021349 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.021352 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.021354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.021475 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.021836 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.021839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.021841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.022084 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.022195 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.022990 (+ 795us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.023208 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 20:38:23.024276 30788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.006904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:38049 (local address 127.23.200.3:48831)
0806 20:38:23.007611 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:23.007654 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:23.009300 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:23.010072 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:23.010080 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:23.010660 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:23.011272 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:23.011314 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.012521 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.012525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:23.014116 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:23.014128 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.015103 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.015108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:23.015313 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.015914 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:23.015935 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:23.018049 (+ 2114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:23.020845 (+ 2796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:23.020851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:23.020862 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:23.021253 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:23.021574 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:23.021577 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:23.021579 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:23.021748 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:23.022184 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:23.022189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:23.022542 (+ 353us) client_negotiation.cc:769] Sending connection context
0806 20:38:23.022864 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 20:38:23.023223 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":277,"threads_started":1}
I20260806 20:38:23.025777 30786 heartbeater.cc:347] Connected to a master server at 127.23.200.62:38049
I20260806 20:38:23.026091 30786 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:23.026950 30786 heartbeater.cc:511] Master 127.23.200.62:38049 requested a full tablet report, sending...
I20260806 20:38:23.028384 30306 ts_manager.cc:194] Registered new tserver with Master: 61a82aebe3d445928b86a28185c856ab (127.23.200.3:41919)
I20260806 20:38:23.029206 30306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:48831
I20260806 20:38:23.039428 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:38:23.056994 30505 heartbeater.cc:503] Master 127.23.200.62:38049 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/test-admin.keytab.
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.067512 24352 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 20:38:23.072523 30794 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:23.070114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41256 (local address 127.23.200.1:38401)
0806 20:38:23.070760 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.070768 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.070792 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:38:23.070849 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.070855 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.071071 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.071199 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.072098 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41256: BlockingRecv error: recv got EOF from 127.0.0.1:41256 (error 108)
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":341,"threads_started":1}
W20260806 20:38:23.072546 24352 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc' of type FILE
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.074282 24352 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.075660 24352 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:23.089919 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.077057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41266 (local address 127.23.200.1:38401)
0806 20:38:23.077257 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.077262 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.077490 (+ 228us) server_negotiation.cc:407] Connection header received
0806 20:38:23.077622 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.077625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.077678 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.077754 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.078716 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.079658 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.080817 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.081352 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.081445 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.084829 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.084838 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.084852 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.084880 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.087726 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.088315 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.088321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.088323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.088434 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.088723 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.088726 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.088728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.089032 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.089246 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.089487 (+ 241us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.089681 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 20:38:23.095407 30794 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:23.093677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41280 (local address 127.23.200.1:38401)
0806 20:38:23.093855 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.093863 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.094149 (+ 286us) server_negotiation.cc:407] Connection header received
0806 20:38:23.094351 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.094355 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.094417 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.094524 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.095175 (+ 651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41280: BlockingRecv error: recv got EOF from 127.0.0.1:41280 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260806 20:38:23.095568 24352 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc' of type FILE
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.097076 24352 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.098276 24352 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:23.111269 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.099566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.23.200.1:38401)
0806 20:38:23.099744 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.099748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.099849 (+ 101us) server_negotiation.cc:407] Connection header received
0806 20:38:23.099953 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.099956 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.100002 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.100074 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.100924 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.101580 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.102773 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.103190 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.103279 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.106606 (+ 3327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.106610 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.106613 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.106640 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.109098 (+ 2458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.109719 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.109723 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.109725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.109832 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.110144 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.110148 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.110149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.110444 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.110640 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.110816 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.110953 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 20:38:23.115340 30794 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:23.113566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41290 (local address 127.23.200.1:38401)
0806 20:38:23.113724 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.113732 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.114099 (+ 367us) server_negotiation.cc:407] Connection header received
0806 20:38:23.114197 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.114200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.114260 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.114393 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.115148 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41290: BlockingRecv error: recv got EOF from 127.0.0.1:41290 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 20:38:23.115530 24352 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786048651485794-24352-0/krb5kdc/krb5cc' of type FILE
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.116894 24352 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:23.118103 24352 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:23.130770 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.119315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.23.200.1:38401)
0806 20:38:23.119516 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.119522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.119749 (+ 227us) server_negotiation.cc:407] Connection header received
0806 20:38:23.119850 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.119854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.119900 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.119972 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.120766 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.121434 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.122649 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.123038 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.123125 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.126153 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.126158 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.126161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.126188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.128664 (+ 2476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.129190 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.129194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.129196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.129310 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.129594 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.129597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.129599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.129833 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.130048 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.130222 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.130429 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:38:23.132184 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30370
I20260806 20:38:23.143344 30794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.132812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41320 (local address 127.23.200.1:38401)
0806 20:38:23.132971 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.132979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.133220 (+ 241us) server_negotiation.cc:407] Connection header received
0806 20:38:23.133392 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.133397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.133453 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.133551 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.134469 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.135116 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.136331 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.136733 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.136871 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.138812 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.138817 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.138820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.138847 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.140904 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.141517 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.141523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.141527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.141674 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.142027 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.142032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.142035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.142266 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.142572 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.142862 (+ 290us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.143061 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 20:38:23.143930 30500 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:23.226820 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30370
I20260806 20:38:23.244534 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30511
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:38:23.260042 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.245515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52180 (local address 127.23.200.2:43719)
0806 20:38:23.246098 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.246106 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.246130 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:38:23.246191 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.246198 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.246558 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.246686 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.247646 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.248754 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.249960 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.250572 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.250666 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.254542 (+ 3876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.254553 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.254565 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.254603 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.257636 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.258237 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.258245 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.258248 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.258426 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.258729 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.258732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.258734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.259054 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.259264 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.259566 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.259755 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":303,"threads_started":1}
I20260806 20:38:23.261266 30641 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:23.343974 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30511
I20260806 20:38:23.361238 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30652
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:38:23.376641 30811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:23.362416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36728 (local address 127.23.200.3:41919)
0806 20:38:23.363128 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:23.363136 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:23.363165 (+ 29us) server_negotiation.cc:407] Connection header received
0806 20:38:23.363225 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:23.363232 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:23.363447 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:23.363594 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:23.364646 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.365758 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.367182 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:23.367738 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:23.367840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:23.371686 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:23.371697 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:23.371710 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:23.371742 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:23.374416 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.375027 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.375034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.375036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.375149 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:23.375463 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:23.375466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:23.375468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:23.375772 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:23.375987 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:23.376173 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 20:38:23.376358 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":406,"threads_started":1}
I20260806 20:38:23.377879 30781 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:23.460202 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30652
I20260806 20:38:23.477588 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30272
I20260806 20:38:23.478814 30337 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:23.581547 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30272
2026-08-06T20:38:23Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3877 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30816](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30816](info): set up 2 sockets
Aug 06 20:38:23 dist-test-slave-tm1g krb5kdc[30816](info): commencing operation
krb5kdc: starting...
W20260806 20:38:24.690256 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:24 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab.
Entry for principal oryx/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab.
I20260806 20:38:24.892050 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:41157
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab
--principal=oryx/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:41157
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:25.209061 30829 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:25.209326 30829 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:25.209506 30829 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:25.216512 30829 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:25.216607 30829 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:25.216652 30829 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:25.216728 30829 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:25.216789 30829 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:25.226586 30829 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:41157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab
--principal=oryx/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:41157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:25.227818 30829 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:25.229117 30829 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:25.242187 30834 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:25.242383 30835 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:25.241958 30837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:25.243582 30829 server_base.cc:1034] running on GCE node
I20260806 20:38:25.244359 30829 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:25.245663 30829 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:25.246866 30829 hybrid_clock.cc:648] HybridClock initialized: now 1786048705246822 us; error 35 us; skew 500 ppm
I20260806 20:38:25.247201 30829 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:25 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048705, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:25.260224 30829 init.cc:377] Logged in from keytab as oryx/127.23.200.62@KRBTEST.COM (short username oryx)
I20260806 20:38:25.261722 30829 webserver.cc:460] Webserver started at http://127.23.200.62:38273/ using document root <none> and password file <none>
I20260806 20:38:25.262480 30829 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:25.262624 30829 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:25.262930 30829 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:25.265913 30829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "53097dc5927343c4ab0d29474bb88790"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "65827414cc48bfd9f8ceb5094e177c19"
server_key_iv: "6fa17897d42ab0e9c56e7dfb28717aff"
server_key_version: "encryptionkey@0"
I20260806 20:38:25.266693 30829 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "53097dc5927343c4ab0d29474bb88790"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "65827414cc48bfd9f8ceb5094e177c19"
server_key_iv: "6fa17897d42ab0e9c56e7dfb28717aff"
server_key_version: "encryptionkey@0"
I20260806 20:38:25.271581 30829 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:38:25.274950 30844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:25.276211 30829 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:38:25.276376 30829 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "53097dc5927343c4ab0d29474bb88790"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "65827414cc48bfd9f8ceb5094e177c19"
server_key_iv: "6fa17897d42ab0e9c56e7dfb28717aff"
server_key_version: "encryptionkey@0"
I20260806 20:38:25.276577 30829 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:25.309681 30829 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:25.313705 30829 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:25.313946 30829 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:25.332710 30829 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:41157
I20260806 20:38:25.332712 30898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:41157 every 8 connection(s)
I20260806 20:38:25.334115 30829 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:25.337707 30899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:25.340867 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30829
I20260806 20:38:25.341102 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:25.341750 24352 external_mini_cluster.cc:1428] Setting key 4fa85e3ee66295f3d2e49f23643d5633
I20260806 20:38:25.350212 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Bootstrap starting.
I20260806 20:38:25.353998 30899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:25.355362 30899 log.cc:824] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Log is configured to *not* fsync() on all Append() calls
Aug 06 20:38:25 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:25.358624 30899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: No bootstrap required, opened a new log
I20260806 20:38:25.364848 30902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:25.343732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54222 (local address 127.23.200.62:41157)
0806 20:38:25.344472 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:25.344482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:25.344536 (+ 54us) server_negotiation.cc:407] Connection header received
0806 20:38:25.346144 (+ 1608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:25.346165 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:25.346616 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:25.346938 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:25.348780 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:25.350075 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:25.351827 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:25.352487 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:25.352613 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:25.357760 (+ 5147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:25.357767 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:25.357780 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:25.357817 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:25.360916 (+ 3099us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:25.361705 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:25.361711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:25.361716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:25.361876 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:25.362283 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:25.362286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:25.362288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:25.362834 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:25.363079 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:25.363509 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 20:38:25.363855 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":356,"threads_started":1}
I20260806 20:38:25.367754 30899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } }
I20260806 20:38:25.368126 30899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:25.368232 30899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53097dc5927343c4ab0d29474bb88790, State: Initialized, Role: FOLLOWER
I20260806 20:38:25.368820 30899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } }
I20260806 20:38:25.369050 30899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:25.369200 30899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:25.369372 30899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:25.371834 30899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } }
I20260806 20:38:25.372298 30899 leader_election.cc:302] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53097dc5927343c4ab0d29474bb88790; no voters:
I20260806 20:38:25.372846 30899 leader_election.cc:288] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:25.372975 30904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:25.373667 30904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 LEADER]: Becoming Leader. State: Replica: 53097dc5927343c4ab0d29474bb88790, State: Running, Role: LEADER
I20260806 20:38:25.374204 30904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } }
I20260806 20:38:25.374930 30899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:25.377272 30906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53097dc5927343c4ab0d29474bb88790. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } }
I20260806 20:38:25.378036 30906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:25.377254 30905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } }
I20260806 20:38:25.378177 30905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:25.383937 30912 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:25.388824 30912 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:25.396651 30912 catalog_manager.cc:1347] Generated new cluster ID: 312d840adb0e4e588bbb6563f1aa42ea
I20260806 20:38:25.396763 30912 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:25.416980 30912 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:25.418402 30912 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:25.425194 30912 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Generated new TSK 0
I20260806 20:38:25.426000 30912 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab.
Entry for principal oryx/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab.
I20260806 20:38:25.485818 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41157
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:25.815268 30925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:25.815531 30925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:25.815624 30925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:25.822911 30925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:25.823009 30925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:25.823112 30925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:25.832203 30925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:25.833331 30925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:25.834676 30925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:25.847812 30930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:25.847779 30931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:25.847892 30933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:25.848471 30925 server_base.cc:1034] running on GCE node
I20260806 20:38:25.849061 30925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:25.849946 30925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:25.851179 30925 hybrid_clock.cc:648] HybridClock initialized: now 1786048705851142 us; error 46 us; skew 500 ppm
I20260806 20:38:25.851493 30925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:25 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048705, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:25.854534 30925 init.cc:377] Logged in from keytab as oryx/127.23.200.1@KRBTEST.COM (short username oryx)
I20260806 20:38:25.855854 30925 webserver.cc:460] Webserver started at http://127.23.200.1:33689/ using document root <none> and password file <none>
I20260806 20:38:25.856554 30925 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:25.856702 30925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:25.857007 30925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:25.859827 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "af44181b22d84f7ab91d75ce7b6a6e0b"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "60837c40879fce5c5afac16631d77fa8"
server_key_iv: "7f3f2e811bdc496f98cc49eee65dcf8c"
server_key_version: "encryptionkey@0"
I20260806 20:38:25.860615 30925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "af44181b22d84f7ab91d75ce7b6a6e0b"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "60837c40879fce5c5afac16631d77fa8"
server_key_iv: "7f3f2e811bdc496f98cc49eee65dcf8c"
server_key_version: "encryptionkey@0"
I20260806 20:38:25.865695 30925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 20:38:25.869107 30940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:25.870370 30925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:25.870545 30925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "af44181b22d84f7ab91d75ce7b6a6e0b"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "60837c40879fce5c5afac16631d77fa8"
server_key_iv: "7f3f2e811bdc496f98cc49eee65dcf8c"
server_key_version: "encryptionkey@0"
I20260806 20:38:25.870738 30925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:25.891072 30925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:25.895282 30925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:25.895562 30925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:25.896442 30925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:25.897704 30925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:25.897814 30925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:25.897959 30925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:25.898039 30925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:25.931321 30925 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:42143
I20260806 20:38:25.931366 31060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:42143 every 8 connection(s)
I20260806 20:38:25.932610 30925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:25.934383 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 30925
I20260806 20:38:25.934639 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:25.935382 24352 external_mini_cluster.cc:1428] Setting key 4aa9566aadb5e47670d0eb4c1bfd5582
Aug 06 20:38:25 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048705, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:25.955152 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:25.935358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:37271 (local address 127.23.200.62:41157)
0806 20:38:25.938655 (+ 3297us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:25.938660 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:25.938675 (+ 15us) server_negotiation.cc:407] Connection header received
0806 20:38:25.939059 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:25.939063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:25.939120 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:25.939201 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:25.941279 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:25.941974 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:25.944500 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:25.945132 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:25.945258 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:25.949276 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:25.949281 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:25.949284 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:25.949329 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:25.951865 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:25.952792 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:25.952796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:25.952798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:25.952909 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:25.953394 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:25.953397 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:25.953398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:25.953645 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:25.953751 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:25.954634 (+ 883us) server_negotiation.cc:299] Negotiation successful
0806 20:38:25.954796 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3193,"thread_start_us":3105,"threads_started":1}
I20260806 20:38:25.956580 31064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:25.935823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:37271)
0806 20:38:25.936645 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:25.936698 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:25.938759 (+ 2061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:25.939516 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:25.939531 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:25.940314 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:25.941109 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:25.941136 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:25.942143 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:25.942151 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:25.944347 (+ 2196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:25.944365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:25.945389 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:25.945399 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:25.945727 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:25.946437 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:25.946464 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:25.949073 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:25.952009 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:25.952020 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:25.952036 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:25.952678 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:25.953017 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:25.953024 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:25.953030 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:25.953265 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:25.953771 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:25.953781 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:25.954214 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 20:38:25.954519 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 20:38:25.955110 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":346,"threads_started":1}
I20260806 20:38:25.958166 31061 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41157
I20260806 20:38:25.958510 31061 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:25.959220 31061 heartbeater.cc:511] Master 127.23.200.62:41157 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:38:25.961251 30862 ts_manager.cc:194] Registered new tserver with Master: af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1:42143)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.2@KRBTEST.COM" created.
I20260806 20:38:25.962873 30862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:37271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab.
Entry for principal oryx/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab.
I20260806 20:38:25.985728 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41157
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:26.306773 31067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:26.307035 31067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:26.307129 31067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:26.314188 31067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:26.314284 31067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:26.314412 31067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:26.324023 31067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:26.325177 31067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:26.326504 31067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:26.339555 31073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:26.339586 31076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:26.342337 31072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:26.343390 31067 server_base.cc:1034] running on GCE node
I20260806 20:38:26.344002 31067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:26.344898 31067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:26.346122 31067 hybrid_clock.cc:648] HybridClock initialized: now 1786048706346076 us; error 40 us; skew 500 ppm
I20260806 20:38:26.346489 31067 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:26 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048706, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:26.349701 31067 init.cc:377] Logged in from keytab as oryx/127.23.200.2@KRBTEST.COM (short username oryx)
I20260806 20:38:26.351150 31067 webserver.cc:460] Webserver started at http://127.23.200.2:36999/ using document root <none> and password file <none>
I20260806 20:38:26.351778 31067 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:26.351873 31067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:26.352106 31067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:26.354897 31067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "47c8d6d62daa401dbbb4d66e3a6377a9"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "716f7013e6523586038b7437fc12f30b"
server_key_iv: "0f76390f738b4de42f0e582c868d0b84"
server_key_version: "encryptionkey@0"
I20260806 20:38:26.355607 31067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "47c8d6d62daa401dbbb4d66e3a6377a9"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "716f7013e6523586038b7437fc12f30b"
server_key_iv: "0f76390f738b4de42f0e582c868d0b84"
server_key_version: "encryptionkey@0"
I20260806 20:38:26.360404 31067 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:38:26.363559 31083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:26.364781 31067 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:26.364910 31067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "47c8d6d62daa401dbbb4d66e3a6377a9"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "716f7013e6523586038b7437fc12f30b"
server_key_iv: "0f76390f738b4de42f0e582c868d0b84"
server_key_version: "encryptionkey@0"
I20260806 20:38:26.365062 31067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:26.410271 31067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:26.414029 31067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:26.414268 31067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:26.415146 31067 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:26.416358 31067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:26.416436 31067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:26.416522 31067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:26.416566 31067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:26.449649 31067 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33665
I20260806 20:38:26.449734 31201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33665 every 8 connection(s)
I20260806 20:38:26.450968 31067 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:26.453853 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 31067
I20260806 20:38:26.454063 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:26.454586 24352 external_mini_cluster.cc:1428] Setting key 5b455a39cc781fac29a15e1dd638d921
Aug 06 20:38:26 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048706, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:26.471029 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:26.453339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:33893 (local address 127.23.200.62:41157)
0806 20:38:26.453507 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:26.453511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:26.454708 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 20:38:26.456565 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:26.456571 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:26.456653 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:26.456763 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:26.458159 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.459332 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.460869 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.461618 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.461764 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:26.464989 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:26.464994 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:26.464997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:26.465030 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:26.467639 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:26.468510 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:26.468525 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:26.468532 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:26.468697 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:26.469134 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:26.469142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:26.469148 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:26.469537 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:26.469699 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:26.470341 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 20:38:26.470604 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 20:38:26.471644 31204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:26.453702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:33893)
0806 20:38:26.454542 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:26.454592 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:26.456329 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:26.456949 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:26.456957 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:26.457452 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:26.457985 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:26.458002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.459474 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.459478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:26.460718 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:26.460730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.461808 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.461813 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:26.462013 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:26.462680 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:26.462701 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:26.464762 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:26.467784 (+ 3022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:26.467792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:26.467806 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:26.468368 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:26.468813 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:26.468816 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:26.468818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:26.469001 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:26.469663 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:26.469680 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:26.469970 (+ 290us) client_negotiation.cc:769] Sending connection context
0806 20:38:26.470183 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 20:38:26.470590 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":384,"threads_started":1}
I20260806 20:38:26.473073 31202 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41157
I20260806 20:38:26.473441 31202 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:26.474403 31202 heartbeater.cc:511] Master 127.23.200.62:41157 requested a full tablet report, sending...
I20260806 20:38:26.476075 30862 ts_manager.cc:194] Registered new tserver with Master: 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665)
WARNING: no policy specified for oryx/127.23.200.3@KRBTEST.COM; defaulting to no policy
I20260806 20:38:26.476866 30862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:33893
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab.
Entry for principal oryx/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab.
I20260806 20:38:26.502943 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41157
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:26.819361 31207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:26.819680 31207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:26.819846 31207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:26.827031 31207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:26.827178 31207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:26.827358 31207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:26.837313 31207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41157
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:26.838650 31207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:26.839932 31207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:26.852761 31212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:26.852864 31215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:26.852806 31213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:26.853623 31207 server_base.cc:1034] running on GCE node
I20260806 20:38:26.854178 31207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:26.855060 31207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:26.856251 31207 hybrid_clock.cc:648] HybridClock initialized: now 1786048706856212 us; error 32 us; skew 500 ppm
I20260806 20:38:26.856544 31207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:26 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048706, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:26.859611 31207 init.cc:377] Logged in from keytab as oryx/127.23.200.3@KRBTEST.COM (short username oryx)
I20260806 20:38:26.860953 31207 webserver.cc:460] Webserver started at http://127.23.200.3:46029/ using document root <none> and password file <none>
I20260806 20:38:26.861617 31207 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:26.861754 31207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:26.862046 31207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:26.864831 31207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "65856518b128475a93a8ffb622c1c1af"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "d36382500336c22f7e3ede150c34291b"
server_key_iv: "5c3bfaae6bc8226c4352670ebf82a0d1"
server_key_version: "encryptionkey@0"
I20260806 20:38:26.865600 31207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "65856518b128475a93a8ffb622c1c1af"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "d36382500336c22f7e3ede150c34291b"
server_key_iv: "5c3bfaae6bc8226c4352670ebf82a0d1"
server_key_version: "encryptionkey@0"
I20260806 20:38:26.870694 31207 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 20:38:26.873817 31222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:26.875155 31207 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:26.875313 31207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "65856518b128475a93a8ffb622c1c1af"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "d36382500336c22f7e3ede150c34291b"
server_key_iv: "5c3bfaae6bc8226c4352670ebf82a0d1"
server_key_version: "encryptionkey@0"
I20260806 20:38:26.875576 31207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:26.907424 31207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:26.911058 31207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:26.911339 31207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:26.912233 31207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:26.913527 31207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:26.913626 31207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:26.913770 31207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:26.913841 31207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:26.946379 31207 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37665
I20260806 20:38:26.946425 31337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37665 every 8 connection(s)
I20260806 20:38:26.947762 31207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:26.951004 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 31207
I20260806 20:38:26.951236 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:26.951809 24352 external_mini_cluster.cc:1428] Setting key f949a87a291ce8055414f43f261e0331
Aug 06 20:38:26 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048706, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:26.965577 31061 heartbeater.cc:503] Master 127.23.200.62:41157 was elected leader, sending a full tablet report...
I20260806 20:38:26.967696 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:26.950358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:40061 (local address 127.23.200.62:41157)
0806 20:38:26.950542 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:26.950547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:26.951737 (+ 1190us) server_negotiation.cc:407] Connection header received
0806 20:38:26.954140 (+ 2403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:26.954145 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:26.954262 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:26.954403 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:26.955871 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.956626 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.958133 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.958589 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.958680 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:26.961828 (+ 3148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:26.961833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:26.961836 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:26.961867 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:26.964471 (+ 2604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:26.965153 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:26.965158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:26.965162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:26.965292 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:26.965849 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:26.965855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:26.965861 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:26.966236 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:26.966386 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:26.967293 (+ 907us) server_negotiation.cc:299] Negotiation successful
0806 20:38:26.967457 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 20:38:26.968786 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:26.950760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:40061)
0806 20:38:26.951564 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:26.951612 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:26.953866 (+ 2254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:26.954649 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:26.954661 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:26.955186 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:26.955722 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:26.955740 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.956766 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.956769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:26.958003 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:26.958014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:26.958764 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:26.958769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:26.958952 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:26.959569 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:26.959594 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:26.961658 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:26.964601 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:26.964607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:26.964616 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:26.965018 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:26.965468 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:26.965473 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:26.965476 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:26.965710 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:26.966420 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:26.966434 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:26.966867 (+ 433us) client_negotiation.cc:769] Sending connection context
0806 20:38:26.967217 (+ 350us) client_negotiation.cc:240] Negotiation successful
0806 20:38:26.967641 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":338,"threads_started":1}
I20260806 20:38:26.970093 31338 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41157
I20260806 20:38:26.970439 31338 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:26.971401 31338 heartbeater.cc:511] Master 127.23.200.62:41157 requested a full tablet report, sending...
I20260806 20:38:26.972846 30862 ts_manager.cc:194] Registered new tserver with Master: 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:26.973711 30862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:40061
I20260806 20:38:26.979015 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.200.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.61.keytab.
Entry for principal oryx/127.23.200.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.61.keytab.
I20260806 20:38:27.023875 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:33647
--webserver_interface=127.23.200.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.61.keytab
--principal=oryx/127.23.200.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:41157,127.23.200.61:33647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:27.345852 31344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:27.346167 31344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:27.346349 31344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:27.353637 31344 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:27.353773 31344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:27.353873 31344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:27.353974 31344 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:27.354050 31344 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:27.363595 31344 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:41157,127.23.200.61:33647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.61.keytab
--principal=oryx/127.23.200.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:33647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.200.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
W20260806 20:38:27.363945 31344 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.200.62:41157,127.23.200.61:33647'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 20:38:27.364933 31344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:27.366192 31344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:27.378067 31352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:27.378119 31350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:27.378010 31349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:27.379313 31344 server_base.cc:1034] running on GCE node
I20260806 20:38:27.380278 31344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:27.381690 31344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:27.382894 31344 hybrid_clock.cc:648] HybridClock initialized: now 1786048707382835 us; error 35 us; skew 500 ppm
I20260806 20:38:27.383203 31344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:27 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048707, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:27.386411 31344 init.cc:377] Logged in from keytab as oryx/127.23.200.61@KRBTEST.COM (short username oryx)
I20260806 20:38:27.387763 31344 webserver.cc:460] Webserver started at http://127.23.200.61:40757/ using document root <none> and password file <none>
I20260806 20:38:27.388429 31344 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:27.388567 31344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:27.388899 31344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:27.391834 31344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/instance:
uuid: "f64bc7e877464bd0af21503023e019c9"
format_stamp: "Formatted at 2026-08-06 20:38:27 on dist-test-slave-tm1g"
server_key: "73a42db2d607fafebf2de1e1251a533c"
server_key_iv: "4979574d364e1eeffa49bcb752dd72b9"
server_key_version: "encryptionkey@0"
I20260806 20:38:27.392608 31344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal/instance:
uuid: "f64bc7e877464bd0af21503023e019c9"
format_stamp: "Formatted at 2026-08-06 20:38:27 on dist-test-slave-tm1g"
server_key: "73a42db2d607fafebf2de1e1251a533c"
server_key_iv: "4979574d364e1eeffa49bcb752dd72b9"
server_key_version: "encryptionkey@0"
I20260806 20:38:27.397568 31344 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 20:38:27.400867 31360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:27.402114 31344 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:38:27.402290 31344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
uuid: "f64bc7e877464bd0af21503023e019c9"
format_stamp: "Formatted at 2026-08-06 20:38:27 on dist-test-slave-tm1g"
server_key: "73a42db2d607fafebf2de1e1251a533c"
server_key_iv: "4979574d364e1eeffa49bcb752dd72b9"
server_key_version: "encryptionkey@0"
I20260806 20:38:27.402582 31344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:27.417158 31344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:27.421298 31344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:27.421571 31344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:27.439889 31344 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.61:33647
I20260806 20:38:27.439920 31415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.61:33647 every 8 connection(s)
I20260806 20:38:27.441190 31344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
I20260806 20:38:27.443596 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 31344
I20260806 20:38:27.444921 31416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:27.454520 31416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:38:27 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
Aug 06 20:38:27 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048707, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.61@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:27.469357 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.445709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53108 (local address 127.23.200.61:33647)
0806 20:38:27.446561 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.446576 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.446624 (+ 48us) server_negotiation.cc:407] Connection header received
0806 20:38:27.448575 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.448600 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.449067 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.449492 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.451453 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.453424 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.455064 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.456044 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.456212 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.460745 (+ 4533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.460753 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.460762 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.460798 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.464460 (+ 3662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.465274 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.465279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.465284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.465456 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.466418 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.466425 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.466430 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.467080 (+ 650us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.467369 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.467741 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.468123 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":349,"threads_started":1}
I20260806 20:38:27.472301 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.457121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54228 (local address 127.23.200.62:41157)
0806 20:38:27.457291 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.457295 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.457826 (+ 531us) server_negotiation.cc:407] Connection header received
0806 20:38:27.457981 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.457988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.458083 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.458203 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.459220 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.460034 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.461552 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.462005 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.462101 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.465911 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.465916 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.465919 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.465949 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.469388 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.470244 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.470250 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.470253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.470438 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.470870 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.470873 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.470875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.471178 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.471284 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.471801 (+ 517us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.472031 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 20:38:27.472422 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.457031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:54228)
0806 20:38:27.457676 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:27.457714 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:27.457861 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:27.458250 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:27.458254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:27.458820 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:27.459061 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.459070 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.460188 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.460194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.461410 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.461421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.462200 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.462209 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.462422 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.462985 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:27.463006 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:27.465748 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:27.469542 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.469556 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.469564 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.470120 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.470558 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.470566 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.470569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.470728 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.471309 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:27.471315 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:27.471484 (+ 169us) client_negotiation.cc:769] Sending connection context
0806 20:38:27.471706 (+ 222us) client_negotiation.cc:240] Negotiation successful
0806 20:38:27.472036 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":295,"threads_started":1}
I20260806 20:38:27.473965 31416 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:38:27.479441 31202 heartbeater.cc:503] Master 127.23.200.62:41157 was elected leader, sending a full tablet report...
Aug 06 20:38:27 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048707, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.61@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
I20260806 20:38:27.486045 31420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.474855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:53122)
0806 20:38:27.475051 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:27.475066 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:27.475185 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:27.475411 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:27.475414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:27.475697 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:27.475816 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.475843 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.476757 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.476761 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.477631 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.477641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.478235 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.478239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.478441 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.479027 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:27.479043 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:27.481329 (+ 2286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:27.484145 (+ 2816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.484152 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.484155 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.484521 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.484855 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.484858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.484861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.484977 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.485423 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:27.485426 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:27.485552 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 20:38:27.485655 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 20:38:27.485834 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 20:38:27.486249 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.474894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.23.200.61:33647)
0806 20:38:27.475092 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.475096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.475129 (+ 33us) server_negotiation.cc:407] Connection header received
0806 20:38:27.475273 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.475277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.475325 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.475429 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.475940 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.476634 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.477743 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.478119 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.478203 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.481479 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.481483 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.481486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.481512 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.484011 (+ 2499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.484617 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.484621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.484623 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.484753 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.485071 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.485074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.485076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.485331 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.485431 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.485741 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.485952 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":92}
I20260806 20:38:27.490605 31416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Bootstrap starting.
I20260806 20:38:27.493450 31416 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:27.494619 31416 log.cc:824] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:27.497385 31416 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: No bootstrap required, opened a new log
I20260806 20:38:27.504222 31416 raft_consensus.cc:348] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } }
I20260806 20:38:27.504511 31416 raft_consensus.cc:374] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:27.504635 31416 raft_consensus.cc:729] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f64bc7e877464bd0af21503023e019c9, State: Initialized, Role: FOLLOWER
I20260806 20:38:27.505227 31416 consensus_queue.cc:260] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } }
I20260806 20:38:27.506150 31422 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } } }
I20260806 20:38:27.506520 31422 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:38:27.507210 31416 sys_catalog.cc:564] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 20:38:27.515877 31433 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:38:27.516012 31433 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:38:27.518447 31344 master_runner.cc:428] Detected that this master f64bc7e877464bd0af21503023e019c9 is joining an existing cluster
I20260806 20:38:27.518550 31344 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.200.61:33647
I20260806 20:38:27.547573 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.522458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54236 (local address 127.23.200.62:41157)
0806 20:38:27.522737 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.522745 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.529365 (+ 6620us) server_negotiation.cc:407] Connection header received
0806 20:38:27.529979 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.529989 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.530106 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.530194 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.532156 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.533654 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.536339 (+ 2685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.536989 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.537098 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.539781 (+ 2683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.539787 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.539793 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.539833 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.543791 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.544776 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.544784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.544789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.544970 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.545561 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.545569 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.545573 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.545989 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.546158 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.546970 (+ 812us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.547209 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260806 20:38:27.547675 31439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.522505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:54236)
0806 20:38:27.529239 (+ 6734us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:27.529264 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:27.529782 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:27.530496 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:27.530502 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:27.531375 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:27.531924 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.531941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.533880 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.533888 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.536168 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.536182 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.537404 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.537410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.537629 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.538331 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:27.538356 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:27.539560 (+ 1204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:27.543975 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.543983 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.543991 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.544619 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.545166 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.545174 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.545181 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.545396 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.546256 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:27.546263 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:27.546555 (+ 292us) client_negotiation.cc:769] Sending connection context
0806 20:38:27.546794 (+ 239us) client_negotiation.cc:240] Negotiation successful
0806 20:38:27.547165 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6551,"thread_start_us":6881,"threads_started":1}
WARNING: no policy specified for oryx/127.23.200.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.60@KRBTEST.COM" created.
I20260806 20:38:27.553478 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.522853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.23.200.61:33647)
0806 20:38:27.523030 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.523036 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.536506 (+ 13470us) server_negotiation.cc:407] Connection header received
0806 20:38:27.536696 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.536703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.536770 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.536832 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.538351 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.539548 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.541664 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.542394 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.542511 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.544868 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.544874 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.544879 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.544910 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.550265 (+ 5355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.551178 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.551184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.551187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.551343 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.551776 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.551780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.551784 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.552134 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.552282 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.552845 (+ 563us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.553085 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260806 20:38:27.554462 31440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.522578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:53130)
0806 20:38:27.536414 (+ 13836us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:27.536436 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:27.536589 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:27.537048 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:27.537055 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:27.537788 (+ 733us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:27.538180 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.538195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.539740 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.539747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.541528 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.541547 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.542760 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.542766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.542934 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.543531 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:27.543551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:27.544694 (+ 1143us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:27.550405 (+ 5711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.550410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.550413 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.551000 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.551492 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.551497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.551500 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.551660 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.552368 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:27.552376 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:27.552549 (+ 173us) client_negotiation.cc:769] Sending connection context
0806 20:38:27.553692 (+ 1143us) client_negotiation.cc:240] Negotiation successful
0806 20:38:27.553978 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13679,"thread_start_us":13512,"threads_started":1}
Aug 06 20:38:27 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048705, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.62@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
I20260806 20:38:27.571645 31419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.556317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53144 (local address 127.23.200.61:33647)
0806 20:38:27.556940 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.556948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.557620 (+ 672us) server_negotiation.cc:407] Connection header received
0806 20:38:27.557952 (+ 332us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.557958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.558101 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.558196 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.559217 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.560225 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.562142 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.562691 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.562840 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.565838 (+ 2998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.565842 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.565846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.565872 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.568703 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.569530 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.569536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.569540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.569668 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.570099 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.570104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.570107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.570495 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.570588 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.571210 (+ 622us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.571379 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476}
I20260806 20:38:27.572793 31442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.555952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:53144)
0806 20:38:27.557482 (+ 1530us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:27.557535 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:27.557748 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:27.558262 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:27.558270 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:27.558871 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:27.559066 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.559075 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.560386 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.560392 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.561993 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.562018 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.562864 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.562870 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.563056 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.563593 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:27.563608 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:27.565658 (+ 2050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:27.568838 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.568850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.568858 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.569402 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.569783 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.569788 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.569792 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.569947 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.570703 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:27.570709 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:27.570843 (+ 134us) client_negotiation.cc:769] Sending connection context
0806 20:38:27.572151 (+ 1308us) client_negotiation.cc:240] Negotiation successful
0806 20:38:27.572467 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1309,"thread_start_us":1171,"threads_started":1}
I20260806 20:38:27.574846 30862 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.200.61:33647: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f64bc7e877464bd0af21503023e019c9"
member_type: NON_VOTER
last_known_addr {
host: "127.23.200.61"
port: 33647
}
attrs {
promote: true
}
}
dest_uuid: "53097dc5927343c4ab0d29474bb88790"
cas_config_opid_index: -1
I20260806 20:38:27.575960 30862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } }
I20260806 20:38:27.579216 31446 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f64bc7e877464bd0af21503023e019c9 (127.23.200.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } }
I20260806 20:38:27.580376 31389 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.60.keytab.
Entry for principal oryx/127.23.200.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.60.keytab.
I20260806 20:38:27.583755 31446 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.200.61:33647
I20260806 20:38:27.584247 31389 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Refusing update from remote peer 53097dc5927343c4ab0d29474bb88790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 20:38:27.584224 31446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } }
I20260806 20:38:27.584414 31446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:27.585207 31344 master.cc:537] Master@127.23.200.61:33647 shutting down...
I20260806 20:38:27.585589 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:45777
--webserver_interface=127.23.200.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.60.keytab
--principal=oryx/127.23.200.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:38:27.586889 31445 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:27.587694 31422 raft_consensus.cc:482] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:27.588800 31422 raft_consensus.cc:504] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } }
I20260806 20:38:27.590092 31422 leader_election.cc:288] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157)
I20260806 20:38:27.590893 30872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f64bc7e877464bd0af21503023e019c9" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53097dc5927343c4ab0d29474bb88790" is_pre_election: true
I20260806 20:38:27.592227 31361 leader_election.cc:302] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: f64bc7e877464bd0af21503023e019c9; no voters: 53097dc5927343c4ab0d29474bb88790
I20260806 20:38:27.593732 31422 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260806 20:38:27.594722 30846 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260806 20:38:27.604362 31344 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 20:38:27.604434 30846 consensus_peers.cc:487] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 -> Peer f64bc7e877464bd0af21503023e019c9 (127.23.200.61:33647): Couldn't send request to peer f64bc7e877464bd0af21503023e019c9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:38:27.604799 31344 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:27.604988 31344 tablet_replica.cc:330] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: stopping tablet replica
I20260806 20:38:27.611572 31344 master.cc:552] Master@127.23.200.61:33647 shutdown complete.
I20260806 20:38:27.611663 31344 master_runner.cc:305] Clearing existing system tablet
I20260806 20:38:27.612149 31344 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:27.612255 31344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:27.615712 31344 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.005s sys 0.000s
I20260806 20:38:27.617619 31451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:27.617939 31344 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:27.618212 31344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260806 20:38:27.618371 31344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
uuid: "f64bc7e877464bd0af21503023e019c9"
format_stamp: "Formatted at 2026-08-06 20:38:27 on dist-test-slave-tm1g"
server_key: "73a42db2d607fafebf2de1e1251a533c"
server_key_iv: "4979574d364e1eeffa49bcb752dd72b9"
server_key_version: "encryptionkey@0"
I20260806 20:38:27.623413 31344 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:27.630084 31344 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 20:38:27.630257 31344 log.cc:1196] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 20:38:27.630693 31344 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Deleting consensus metadata
I20260806 20:38:27.630976 31344 master_runner.cc:315] Copying system tablet from 127.23.200.62:41157
I20260806 20:38:27.633447 31344 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: tablet copy: Beginning tablet copy session from remote peer at address 127.23.200.62:41157
I20260806 20:38:27.654881 31457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.634246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:54242)
0806 20:38:27.640319 (+ 6073us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:27.640342 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:27.640493 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:27.640955 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:27.640961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:27.641451 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:27.641727 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.641738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.643216 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.643222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.644800 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:27.644815 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.645836 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.645842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:27.646032 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.646731 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:27.646754 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:27.647826 (+ 1072us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:27.651781 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.651786 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.651790 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.652255 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.652755 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:27.652760 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:27.652764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:27.652936 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:27.653691 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:27.653697 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:27.653864 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 20:38:27.654022 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 20:38:27.654367 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5923,"thread_start_us":5819,"threads_started":1}
I20260806 20:38:27.654947 31063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:27.634294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54242 (local address 127.23.200.62:41157)
0806 20:38:27.634495 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:27.634510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:27.640459 (+ 5949us) server_negotiation.cc:407] Connection header received
0806 20:38:27.640683 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:27.640689 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:27.640790 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:27.640894 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:27.641897 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.643068 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.644978 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:27.645679 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:27.645803 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:27.648081 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:27.648090 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:27.648097 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:27.648144 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:27.651639 (+ 3495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.652424 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.652433 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.652439 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.652616 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:27.653073 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:27.653079 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:27.653083 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:27.653554 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:27.653737 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:27.654188 (+ 451us) server_negotiation.cc:299] Negotiation successful
0806 20:38:27.654504 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260806 20:38:27.656280 30882 tablet_copy_service.cc:140] P 53097dc5927343c4ab0d29474bb88790: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f64bc7e877464bd0af21503023e019c9 ({username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:54242)
I20260806 20:38:27.656597 30882 tablet_copy_service.cc:161] P 53097dc5927343c4ab0d29474bb88790: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f64bc7e877464bd0af21503023e019c9 at {username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:54242: session id = f64bc7e877464bd0af21503023e019c9-00000000000000000000000000000000
I20260806 20:38:27.661242 30882 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:38:27.663741 31344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 20:38:27.670881 31344 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: tablet copy: Starting download of 0 data blocks...
I20260806 20:38:27.671259 31344 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: tablet copy: Starting download of 1 WAL segments...
I20260806 20:38:27.674347 31344 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:38:27.678987 30882 tablet_copy_service.cc:342] P 53097dc5927343c4ab0d29474bb88790: Request end of tablet copy session f64bc7e877464bd0af21503023e019c9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:54242
I20260806 20:38:27.679281 30882 tablet_copy_service.cc:433] P 53097dc5927343c4ab0d29474bb88790: ending tablet copy session f64bc7e877464bd0af21503023e019c9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f64bc7e877464bd0af21503023e019c9
I20260806 20:38:27.692260 31344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:27.695101 31458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:27.699563 31459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:27.704769 31344 server_base.cc:1034] running on GCE node
W20260806 20:38:27.704775 31461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:27.705507 31344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:27.706274 31344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:27.707564 31344 hybrid_clock.cc:648] HybridClock initialized: now 1786048707707488 us; error 48 us; skew 500 ppm
I20260806 20:38:27.707790 31344 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:27 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048707, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:27.709961 31344 init.cc:377] Logged in from keytab as oryx/127.23.200.61@KRBTEST.COM (short username oryx)
I20260806 20:38:27.715349 31344 webserver.cc:460] Webserver started at http://127.23.200.61:43355/ using document root <none> and password file <none>
I20260806 20:38:27.715723 31344 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:27.715802 31344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:27.719085 31344 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 20:38:27.721099 31468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:27.721554 31344 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260806 20:38:27.721689 31344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
uuid: "f64bc7e877464bd0af21503023e019c9"
format_stamp: "Formatted at 2026-08-06 20:38:27 on dist-test-slave-tm1g"
server_key: "73a42db2d607fafebf2de1e1251a533c"
server_key_iv: "4979574d364e1eeffa49bcb752dd72b9"
server_key_version: "encryptionkey@0"
I20260806 20:38:27.721848 31344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:27.751837 31344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:27.755897 31344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:27.786628 31344 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.61:33647
I20260806 20:38:27.786725 31523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.61:33647 every 8 connection(s)
I20260806 20:38:27.787720 31344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
I20260806 20:38:27.793272 31524 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:38:27.798239 31524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Bootstrap starting.
I20260806 20:38:27.816077 31524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:38:27.816787 31524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Bootstrap complete.
I20260806 20:38:27.818043 31524 raft_consensus.cc:348] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } }
I20260806 20:38:27.818418 31524 raft_consensus.cc:729] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f64bc7e877464bd0af21503023e019c9, State: Initialized, Role: LEARNER
I20260806 20:38:27.818723 31524 consensus_queue.cc:260] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } }
I20260806 20:38:27.819572 31527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } }
I20260806 20:38:27.819872 31527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: LEARNER
I20260806 20:38:27.819976 31524 sys_catalog.cc:564] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:27.827677 31538 catalog_manager.cc:1259] Loaded cluster ID: 312d840adb0e4e588bbb6563f1aa42ea
I20260806 20:38:27.827786 31538 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: loading cluster ID for follower catalog manager: success
I20260806 20:38:27.830929 31538 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: acquiring CA information for follower catalog manager: success
I20260806 20:38:27.833043 31538 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 20:38:27.976365 31338 heartbeater.cc:503] Master 127.23.200.62:41157 was elected leader, sending a full tablet report...
W20260806 20:38:28.035844 31449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:28.036172 31449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:28.036338 31449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:28.044049 31449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:28.044189 31449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:28.044283 31449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:28.044385 31449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:28.044459 31449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:28.054169 31449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.60.keytab
--principal=oryx/127.23.200.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:45777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.200.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:28.055516 31449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:28.056784 31449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:28.069057 31543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:28.069128 31544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:28.069022 31546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:28.075121 31449 server_base.cc:1034] running on GCE node
I20260806 20:38:28.075927 31449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:28.077288 31449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:28.078481 31449 hybrid_clock.cc:648] HybridClock initialized: now 1786048708078430 us; error 39 us; skew 500 ppm
I20260806 20:38:28.078835 31449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:28.082414 31449 init.cc:377] Logged in from keytab as oryx/127.23.200.60@KRBTEST.COM (short username oryx)
I20260806 20:38:28.083940 31449 webserver.cc:460] Webserver started at http://127.23.200.60:34445/ using document root <none> and password file <none>
I20260806 20:38:28.084667 31449 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:28.084798 31449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:28.085105 31449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:28.088052 31449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/instance:
uuid: "f49924b2bf0c4ac999c321d80fbcd397"
format_stamp: "Formatted at 2026-08-06 20:38:28 on dist-test-slave-tm1g"
server_key: "3c279a0224231b2910761a06486d8ae8"
server_key_iv: "c1c5505973ce3856d8cf974c3a007cde"
server_key_version: "encryptionkey@0"
I20260806 20:38:28.088802 31449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal/instance:
uuid: "f49924b2bf0c4ac999c321d80fbcd397"
format_stamp: "Formatted at 2026-08-06 20:38:28 on dist-test-slave-tm1g"
server_key: "3c279a0224231b2910761a06486d8ae8"
server_key_iv: "c1c5505973ce3856d8cf974c3a007cde"
server_key_version: "encryptionkey@0"
I20260806 20:38:28.093712 31449 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:38:28.097221 31553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:28.098520 31449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:28.098691 31449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
uuid: "f49924b2bf0c4ac999c321d80fbcd397"
format_stamp: "Formatted at 2026-08-06 20:38:28 on dist-test-slave-tm1g"
server_key: "3c279a0224231b2910761a06486d8ae8"
server_key_iv: "c1c5505973ce3856d8cf974c3a007cde"
server_key_version: "encryptionkey@0"
I20260806 20:38:28.098878 31449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:28.114499 31449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:28.119269 31449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:28.119658 31449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:28.147984 31449 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.60:45777
I20260806 20:38:28.148073 31606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.60:45777 every 8 connection(s)
I20260806 20:38:28.149349 31449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
I20260806 20:38:28.152999 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 31449
I20260806 20:38:28.153746 31607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:28.162546 31607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.60@KRBTEST.COM
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.60@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:28.181272 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.155184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57624 (local address 127.23.200.60:45777)
0806 20:38:28.155954 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.155964 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.156004 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:38:28.158610 (+ 2606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.158630 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.159043 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.159448 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.160939 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.162351 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.163806 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.164473 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.164678 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.170222 (+ 5544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.170230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.170239 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.170291 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.174282 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.175531 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.175537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.175542 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.175798 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.176343 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.176351 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.176354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.177176 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.177498 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.178817 (+ 1319us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.179259 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":354,"threads_started":1}
I20260806 20:38:28.184777 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.165116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54252 (local address 127.23.200.62:41157)
0806 20:38:28.166390 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.166394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.166410 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:38:28.166460 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.166464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.166521 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.166787 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.167837 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.168601 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.170286 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.170750 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.170898 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.174819 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.174825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.174831 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.174883 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.180511 (+ 5628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.181727 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.181732 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.181736 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.181988 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.182703 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.182708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.182711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.183096 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.183289 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.183909 (+ 620us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.184146 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1117,"thread_start_us":1031,"threads_started":1}
I20260806 20:38:28.185334 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.165015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:54252)
0806 20:38:28.165765 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.165804 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.165953 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.166852 (+ 899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.166858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.167428 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.167680 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.167688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.168784 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.168791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.170060 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.170073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.170949 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.170956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.171135 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.171873 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.171890 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.174628 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.180720 (+ 6092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.180730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.180736 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.181582 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.182177 (+ 595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.182185 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.182191 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.182535 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.183241 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.183248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.183553 (+ 305us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.183805 (+ 252us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.184146 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":389,"threads_started":1}
I20260806 20:38:28.186793 31607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:38:28.194895 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.179080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:53158)
0806 20:38:28.179996 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.180081 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.180254 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.187332 (+ 7078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.187337 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.187396 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:28.187876 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.187886 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.190356 (+ 2470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.190362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.191995 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.192008 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.193584 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.193590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.193892 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.194042 (+ 150us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.194181 (+ 139us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.194392 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":587,"threads_started":1}
I20260806 20:38:28.195243 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.179797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.23.200.61:33647)
0806 20:38:28.187003 (+ 7206us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.187010 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.187031 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:38:28.187101 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.187105 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.187171 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.187297 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:28.188032 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.190165 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.192202 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.193322 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.193557 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.193863 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.194208 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.194431 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":7035,"thread_start_us":6927,"threads_started":1}
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.60@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
I20260806 20:38:28.196518 31498 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 20:38:28.197151 31527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53097dc5927343c4ab0d29474bb88790. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } }
I20260806 20:38:28.197436 31527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: LEARNER
I20260806 20:38:28.202852 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.187955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:53164)
0806 20:38:28.188352 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.188372 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.188493 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.189556 (+ 1063us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.189559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.190018 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.190247 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.190255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.191516 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.191523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.192618 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.192629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.193465 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.193470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.193673 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.194252 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.194269 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.196690 (+ 2421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.200331 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.200338 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.200342 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.200771 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.201153 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.201157 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.201161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.201345 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.201863 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.201868 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.202033 (+ 165us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.202169 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.202519 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257}
I20260806 20:38:28.202968 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.187957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53164 (local address 127.23.200.61:33647)
0806 20:38:28.189254 (+ 1297us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.189260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.189283 (+ 23us) server_negotiation.cc:407] Connection header received
0806 20:38:28.189327 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.189332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.189400 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.189506 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.190493 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.191387 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.192755 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.193296 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.193456 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.196972 (+ 3516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.196979 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.196983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.197021 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.200189 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.200896 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.200903 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.200911 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.201042 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.201453 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.201460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.201463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.201749 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.201896 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.202255 (+ 359us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.202531 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":1107,"thread_start_us":861,"threads_started":1}
I20260806 20:38:28.204043 31607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.60@KRBTEST.COM for oryx/127.23.200.60@KRBTEST.COM
I20260806 20:38:28.216854 31611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.204667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.0.0.1:57636)
0806 20:38:28.204907 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.204922 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.205010 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.205235 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.205238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.205616 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.205774 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.205782 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.206780 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.206784 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.207608 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.207618 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.208327 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.208331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.208563 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.209121 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.209136 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.211529 (+ 2393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.214716 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.214719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.214722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.215132 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.215432 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.215435 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.215437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.215639 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.216082 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.216085 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.216198 (+ 113us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.216300 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.216467 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 20:38:28.216868 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.204718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57636 (local address 127.23.200.60:45777)
0806 20:38:28.204899 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.204903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.204981 (+ 78us) server_negotiation.cc:407] Connection header received
0806 20:38:28.205078 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.205081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.205131 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.205202 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.205891 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.206674 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.207724 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.208193 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.208289 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.211693 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.211697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.211700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.211728 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.214610 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.215226 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.215229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.215231 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.215333 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.215727 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.215730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.215732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.215989 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.216091 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.216391 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.216587 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":47}
I20260806 20:38:28.222008 31607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Bootstrap starting.
I20260806 20:38:28.225615 31607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:28.227030 31607 log.cc:824] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:28.229734 31607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: No bootstrap required, opened a new log
I20260806 20:38:28.238973 31607 raft_consensus.cc:348] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } }
I20260806 20:38:28.239341 31607 raft_consensus.cc:374] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:28.239491 31607 raft_consensus.cc:729] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f49924b2bf0c4ac999c321d80fbcd397, State: Initialized, Role: FOLLOWER
I20260806 20:38:28.240083 31607 consensus_queue.cc:260] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } }
I20260806 20:38:28.241284 31619 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } } }
I20260806 20:38:28.241961 31619 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:38:28.242237 31607 sys_catalog.cc:564] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 20:38:28.258391 31630 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:38:28.258606 31630 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:38:28.263033 31449 master_runner.cc:428] Detected that this master f49924b2bf0c4ac999c321d80fbcd397 is joining an existing cluster
I20260806 20:38:28.263228 31449 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.200.60:45777
I20260806 20:38:28.288353 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30925
I20260806 20:38:28.290795 31639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.269073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:53174)
0806 20:38:28.271685 (+ 2612us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.271981 (+ 296us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.272161 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.272697 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.272702 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.273644 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.274009 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.274021 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.275604 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.275609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.277687 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.277701 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.278959 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.278966 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.279144 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.280009 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.280033 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.281817 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.287303 (+ 5486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.287307 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.287310 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.287770 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.288210 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.288215 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.288218 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.288602 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.289398 (+ 796us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.289405 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.289661 (+ 256us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.289816 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.290129 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2456,"mutex_wait_us":1,"thread_start_us":2282,"threads_started":1}
I20260806 20:38:28.292173 31617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.269257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.23.200.61:33647)
0806 20:38:28.269520 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.269529 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.272063 (+ 2534us) server_negotiation.cc:407] Connection header received
0806 20:38:28.272336 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.272343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.272431 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.272504 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.274183 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.275338 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.277871 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.278536 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.278677 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.282021 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.282027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.282032 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.282070 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.287143 (+ 5073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.287908 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.287913 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.287917 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.288072 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.288746 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.288751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.288753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.289113 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.289304 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.291396 (+ 2092us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.291704 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 20:38:28.292194 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.268783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54264 (local address 127.23.200.62:41157)
0806 20:38:28.269477 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.269484 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.271069 (+ 1585us) server_negotiation.cc:407] Connection header received
0806 20:38:28.271232 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.271241 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.271379 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.271485 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.273976 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.275530 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.278649 (+ 3119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.279605 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.279730 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.283279 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.283289 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.283296 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.283390 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.287961 (+ 4571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.289005 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.289013 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.289019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.289226 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.289837 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.289846 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.289852 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.290362 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.290535 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.291352 (+ 817us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.291678 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559}
I20260806 20:38:28.293761 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.269031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:54264)
0806 20:38:28.270938 (+ 1907us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.270959 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.271098 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.271710 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.271715 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.273245 (+ 1530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.273744 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.273769 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.275878 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.275885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.278434 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.278454 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.280211 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.280221 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.280518 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.281274 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.281323 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.282983 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.288162 (+ 5179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.288170 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.288177 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.288836 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.289419 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.289426 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.289432 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.289686 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.290649 (+ 963us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.290656 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.290968 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.292507 (+ 1539us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.293011 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1762,"thread_start_us":1607,"threads_started":1}
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.1@KRBTEST.COM
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048705, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.62@KRBTEST.COM for oryx/127.23.200.60@KRBTEST.COM
I20260806 20:38:28.317698 31610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.272560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.23.200.60:45777)
0806 20:38:28.272758 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.272764 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.295705 (+ 22941us) server_negotiation.cc:407] Connection header received
0806 20:38:28.295943 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.295950 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.296038 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.296119 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.298418 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.300256 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.303284 (+ 3028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.304102 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.304199 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.307270 (+ 3071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.307278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.307284 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.307329 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.313406 (+ 6077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.314435 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.314449 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.314452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.314661 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.315160 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.315165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.315170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.315789 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.315951 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.316742 (+ 791us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.317003 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":56}
I20260806 20:38:28.319195 31638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.272730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.0.0.1:57652)
0806 20:38:28.295609 (+ 22879us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.295633 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.295803 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.296371 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.296378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.297105 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.298227 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.298243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.301251 (+ 3008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.301256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.303125 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.303139 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.304503 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.304509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.304704 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.305561 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.305592 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.307082 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.313534 (+ 6452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.313542 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.313546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.314298 (+ 752us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.314797 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.314801 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.314805 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.315031 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.316084 (+ 1053us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.316091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.316290 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.318271 (+ 1981us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.318668 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22699,"thread_start_us":24858,"threads_started":1}
I20260806 20:38:28.323911 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.300071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.0.0.1:57656)
0806 20:38:28.300469 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.300484 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.300567 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.302032 (+ 1465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.302038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.302648 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.302900 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.302928 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.305402 (+ 2474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.305408 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.307012 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.307028 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.308632 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.308639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.308827 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.309516 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.309545 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.314470 (+ 4925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.320097 (+ 5627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.320104 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.320107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.320828 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.321352 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.321357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.321360 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.321536 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.322753 (+ 1217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.322758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.322981 (+ 223us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.323114 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.323414 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260806 20:38:28.324021 31641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.290136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.23.200.1:42143)
0806 20:38:28.298153 (+ 8017us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.298162 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.298192 (+ 30us) server_negotiation.cc:407] Connection header received
0806 20:38:28.298261 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.298270 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.298585 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.300525 (+ 1940us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.300855 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.303131 (+ 2276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.305173 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.306334 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.310428 (+ 4094us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.314255 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.314279 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.314299 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.314367 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.319016 (+ 4649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.319978 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.319986 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.319990 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.320164 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.320883 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.320888 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.320892 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.321331 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.322668 (+ 1337us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.322993 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.323344 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7821,"thread_start_us":8162,"threads_started":1}
I20260806 20:38:28.325851 31056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:28.326225 31642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.300146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57656 (local address 127.23.200.60:45777)
0806 20:38:28.301594 (+ 1448us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.301600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.301627 (+ 27us) server_negotiation.cc:407] Connection header received
0806 20:38:28.301740 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.301745 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.301848 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.303069 (+ 1221us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.303458 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.304882 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.307212 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.308372 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.309148 (+ 776us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.314817 (+ 5669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.314824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.314828 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.314881 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.319948 (+ 5067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.321005 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.321009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.321012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.321215 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.322110 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.322114 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.322123 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.322601 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.325252 (+ 2651us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.325458 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.325674 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":1292,"thread_start_us":929,"threads_started":1}
I20260806 20:38:28.327618 30862 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.200.60:45777: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397"
member_type: NON_VOTER
last_known_addr {
host: "127.23.200.60"
port: 45777
}
attrs {
promote: true
}
}
dest_uuid: "53097dc5927343c4ab0d29474bb88790"
cas_config_opid_index: 5
I20260806 20:38:28.328480 30862 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } }
I20260806 20:38:28.331200 31498 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 LEARNER]: Refusing update from remote peer 53097dc5927343c4ab0d29474bb88790: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 20:38:28.330860 31613 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.331614 31582 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:28.336048 31582 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Refusing update from remote peer 53097dc5927343c4ab0d29474bb88790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 20:38:28.355104 31646 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260806 20:38:28.367882 31498 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
W20260806 20:38:28.372466 31560 tablet.cc:2369] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:38:28.374123 31644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:28.375813 31613 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.200.60:45777
I20260806 20:38:28.376685 31449 master.cc:537] Master@127.23.200.60:45777 shutting down...
I20260806 20:38:28.376513 31527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.376962 31527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: LEARNER
I20260806 20:38:28.387181 31646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.387622 31646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:28.388245 31613 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: attempting to promote NON_VOTER f64bc7e877464bd0af21503023e019c9 to VOTER
I20260806 20:38:28.389099 31613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } }
I20260806 20:38:28.392606 31648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: NON_VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: true } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } } pending_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.393194 31648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:28.393989 31619 raft_consensus.cc:482] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:28.394228 31619 raft_consensus.cc:504] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } }
I20260806 20:38:28.394941 31498 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 LEARNER]: Refusing update from remote peer 53097dc5927343c4ab0d29474bb88790: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 20:38:28.395663 31644 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 20:38:28.395807 31619 leader_election.cc:288] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157), f64bc7e877464bd0af21503023e019c9 (127.23.200.61:33647)
I20260806 20:38:28.396533 30872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f49924b2bf0c4ac999c321d80fbcd397" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53097dc5927343c4ab0d29474bb88790" is_pre_election: true
I20260806 20:38:28.396534 31498 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f49924b2bf0c4ac999c321d80fbcd397" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f64bc7e877464bd0af21503023e019c9" is_pre_election: true
I20260806 20:38:28.397466 31554 leader_election.cc:302] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f49924b2bf0c4ac999c321d80fbcd397; no voters: 53097dc5927343c4ab0d29474bb88790, f64bc7e877464bd0af21503023e019c9
I20260806 20:38:28.398046 31619 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:38:28.400306 31645 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f64bc7e877464bd0af21503023e019c9 (127.23.200.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.400987 31496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f64bc7e877464bd0af21503023e019c9 (127.23.200.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.405290 31527 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.405548 31527 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:38:28.408052 31449 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 20:38:28.408377 30848 consensus_peers.cc:487] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 -> Peer f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60:45777): Couldn't send request to peer f49924b2bf0c4ac999c321d80fbcd397. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 20:38:28.408454 31449 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:28.408638 31449 tablet_replica.cc:330] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: stopping tablet replica
I20260806 20:38:28.411906 31644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.412160 31644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:28.413204 31613 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.413407 31613 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:28.416757 31449 master.cc:552] Master@127.23.200.60:45777 shutdown complete.
I20260806 20:38:28.416854 31449 master_runner.cc:305] Clearing existing system tablet
I20260806 20:38:28.417197 31449 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:28.417312 31449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:28.419693 31449 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 20:38:28.421377 31652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:28.421784 31449 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:28.422091 31449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260806 20:38:28.422276 31449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
uuid: "f49924b2bf0c4ac999c321d80fbcd397"
format_stamp: "Formatted at 2026-08-06 20:38:28 on dist-test-slave-tm1g"
server_key: "3c279a0224231b2910761a06486d8ae8"
server_key_iv: "c1c5505973ce3856d8cf974c3a007cde"
server_key_version: "encryptionkey@0"
I20260806 20:38:28.426148 31449 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:28.430939 31449 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 20:38:28.431105 31449 log.cc:1196] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 20:38:28.431555 31449 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Deleting consensus metadata
I20260806 20:38:28.431803 31449 master_runner.cc:315] Copying system tablet from 127.23.200.62:41157
I20260806 20:38:28.434432 31449 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: tablet copy: Beginning tablet copy session from remote peer at address 127.23.200.62:41157
I20260806 20:38:28.448235 31612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.435500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57568 (local address 127.23.200.62:41157)
0806 20:38:28.435737 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.435744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.435945 (+ 201us) server_negotiation.cc:407] Connection header received
0806 20:38:28.436185 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.436191 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.436283 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.436391 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:28.437310 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.438052 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.439387 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.439839 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.439988 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.441975 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:28.441981 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:28.441985 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:28.442023 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:28.445268 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.445958 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.445964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.445967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.446118 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:28.446620 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:28.446628 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:28.446632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:28.446944 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:28.447103 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.447581 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.447845 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 20:38:28.448251 31660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.435161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:57568)
0806 20:38:28.435829 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.435851 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.436052 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.436428 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.436432 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.436905 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:28.437159 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.437169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.438208 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.438213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.439225 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.439236 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.440008 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.440014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.440199 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.440870 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:28.440888 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:28.441744 (+ 856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:28.445424 (+ 3680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.445431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.445436 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.445802 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.446277 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:28.446280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:28.446282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:28.446499 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:28.447067 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:28.447072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:28.447270 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.447450 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.447778 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":455,"threads_started":1}
I20260806 20:38:28.451701 30882 tablet_copy_service.cc:140] P 53097dc5927343c4ab0d29474bb88790: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f49924b2bf0c4ac999c321d80fbcd397 ({username='oryx', principal='oryx/127.23.200.60@KRBTEST.COM'} at 127.0.0.1:57568)
I20260806 20:38:28.451908 30882 tablet_copy_service.cc:161] P 53097dc5927343c4ab0d29474bb88790: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f49924b2bf0c4ac999c321d80fbcd397 at {username='oryx', principal='oryx/127.23.200.60@KRBTEST.COM'} at 127.0.0.1:57568: session id = f49924b2bf0c4ac999c321d80fbcd397-00000000000000000000000000000000
I20260806 20:38:28.454417 30882 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 20:38:28.456018 31449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 20:38:28.461000 31449 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: tablet copy: Starting download of 0 data blocks...
I20260806 20:38:28.461278 31449 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: tablet copy: Starting download of 1 WAL segments...
I20260806 20:38:28.464082 31449 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 20:38:28.467613 30882 tablet_copy_service.cc:342] P 53097dc5927343c4ab0d29474bb88790: Request end of tablet copy session f49924b2bf0c4ac999c321d80fbcd397-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.200.60@KRBTEST.COM'} at 127.0.0.1:57568
I20260806 20:38:28.467767 30882 tablet_copy_service.cc:433] P 53097dc5927343c4ab0d29474bb88790: ending tablet copy session f49924b2bf0c4ac999c321d80fbcd397-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f49924b2bf0c4ac999c321d80fbcd397
I20260806 20:38:28.480738 31449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:28.484474 31661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:28.497838 31662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:28.498426 31449 server_base.cc:1034] running on GCE node
W20260806 20:38:28.498888 31664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:28.498997 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30925
I20260806 20:38:28.499460 31449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:28.500164 31449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:28.501570 31449 hybrid_clock.cc:648] HybridClock initialized: now 1786048708501451 us; error 51 us; skew 500 ppm
I20260806 20:38:28.501940 31449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:28.504362 31449 init.cc:377] Logged in from keytab as oryx/127.23.200.60@KRBTEST.COM (short username oryx)
I20260806 20:38:28.515977 31449 webserver.cc:460] Webserver started at http://127.23.200.60:40521/ using document root <none> and password file <none>
I20260806 20:38:28.516528 31449 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:28.516701 31449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:28.519364 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:42143
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:41157
--webserver_port=33689
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:38:28.520004 31449 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260806 20:38:28.522133 31671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:28.522919 31449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260806 20:38:28.523066 31449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
uuid: "f49924b2bf0c4ac999c321d80fbcd397"
format_stamp: "Formatted at 2026-08-06 20:38:28 on dist-test-slave-tm1g"
server_key: "3c279a0224231b2910761a06486d8ae8"
server_key_iv: "c1c5505973ce3856d8cf974c3a007cde"
server_key_version: "encryptionkey@0"
I20260806 20:38:28.523252 31449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:28.540793 31449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:28.544890 31449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:28.566346 31449 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.60:45777
I20260806 20:38:28.566488 31727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.60:45777 every 8 connection(s)
I20260806 20:38:28.567353 31449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
I20260806 20:38:28.573537 31728 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:38:28.577562 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Bootstrap starting.
I20260806 20:38:28.597404 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:38:28.597944 31728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: Bootstrap complete.
I20260806 20:38:28.599043 31728 raft_consensus.cc:348] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } }
I20260806 20:38:28.599372 31728 raft_consensus.cc:729] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f49924b2bf0c4ac999c321d80fbcd397, State: Initialized, Role: LEARNER
I20260806 20:38:28.599638 31728 consensus_queue.cc:260] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } }
I20260806 20:38:28.600670 31728 sys_catalog.cc:564] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:28.600463 31732 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.601042 31732 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: This master's current role is: LEARNER
I20260806 20:38:28.609258 31743 catalog_manager.cc:1259] Loaded cluster ID: 312d840adb0e4e588bbb6563f1aa42ea
I20260806 20:38:28.609351 31743 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: loading cluster ID for follower catalog manager: success
I20260806 20:38:28.612699 31743 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: acquiring CA information for follower catalog manager: success
I20260806 20:38:28.614998 31743 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 20:38:28.851907 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.844143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47894 (local address 127.23.200.60:45777)
0806 20:38:28.845245 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:28.845249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:28.845265 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:38:28.845339 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:28.845343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:28.845411 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:28.845507 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:28.846124 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.847229 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.849684 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.850576 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.850673 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.850852 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:28.851315 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 20:38:28.851480 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":999,"thread_start_us":827,"threads_started":1}
I20260806 20:38:28.851995 31744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:28.843924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.0.0.1:47894)
0806 20:38:28.844464 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:28.844493 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:28.844695 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:28.845591 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:28.845599 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:28.845635 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:28.845955 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.845971 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.847409 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.847414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.849517 (+ 2103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:28.849536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:28.850765 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:28.850772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:28.851094 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:28.851141 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 20:38:28.851284 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 20:38:28.851524 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":252,"threads_started":1}
I20260806 20:38:28.853267 31700 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 20:38:28.854007 31732 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53097dc5927343c4ab0d29474bb88790. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:28.854362 31732 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: This master's current role is: LEARNER
W20260806 20:38:28.902447 31673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:28.902804 31673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:28.902951 31673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:28.911676 31673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:28.911835 31673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:28.911991 31673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:28.923154 31673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:42143
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=33689
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:28.924808 31673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:28.926205 31673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:28.941570 31751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:28.941550 31752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:28.942119 31754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:28.943094 31673 server_base.cc:1034] running on GCE node
I20260806 20:38:28.943790 31673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:28.944792 31673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:28.946146 31673 hybrid_clock.cc:648] HybridClock initialized: now 1786048708946031 us; error 61 us; skew 500 ppm
I20260806 20:38:28.946511 31673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:28 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:28.950038 31673 init.cc:377] Logged in from keytab as oryx/127.23.200.1@KRBTEST.COM (short username oryx)
I20260806 20:38:28.951668 31673 webserver.cc:460] Webserver started at http://127.23.200.1:33689/ using document root <none> and password file <none>
I20260806 20:38:28.952394 31673 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:28.952564 31673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:28.958438 31673 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 20:38:28.961040 31761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:28.962570 31673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:28.962776 31673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "af44181b22d84f7ab91d75ce7b6a6e0b"
format_stamp: "Formatted at 2026-08-06 20:38:25 on dist-test-slave-tm1g"
server_key: "60837c40879fce5c5afac16631d77fa8"
server_key_iv: "7f3f2e811bdc496f98cc49eee65dcf8c"
server_key_version: "encryptionkey@0"
I20260806 20:38:28.963794 31673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:29.013051 31673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:29.017570 31673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:29.017933 31673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:29.019155 31673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:29.020772 31673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:29.020929 31673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:29.021104 31673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:29.021248 31673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:29.071534 31673 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:42143
I20260806 20:38:29.071621 31882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:42143 every 8 connection(s)
I20260806 20:38:29.073032 31673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:29.074259 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 31673
I20260806 20:38:29.074625 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 31067
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.2@KRBTEST.COM
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.60@KRBTEST.COM
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048708, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
I20260806 20:38:29.120344 31889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.078257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.23.200.2:33665)
0806 20:38:29.089430 (+ 11173us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.089442 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.090254 (+ 812us) server_negotiation.cc:407] Connection header received
0806 20:38:29.090548 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.090558 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.090803 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.090951 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.093787 (+ 2836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.095678 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.098384 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.099665 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.099830 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.109712 (+ 9882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.109730 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.109748 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.109819 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.115152 (+ 5333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.116950 (+ 1798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.116962 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.116966 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.117203 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.117636 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.117642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.117655 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.118132 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.118477 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.119473 (+ 996us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.119751 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10951,"thread_start_us":11172,"threads_started":1}
I20260806 20:38:29.122375 31197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:29.124518 31745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.076366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:59165 (local address 127.23.200.60:45777)
0806 20:38:29.076620 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.076629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.090925 (+ 14296us) server_negotiation.cc:407] Connection header received
0806 20:38:29.091076 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.091084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.091187 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.091305 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.098490 (+ 7185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.100040 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.105471 (+ 5431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.106331 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.106471 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.113888 (+ 7417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.113895 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.113899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.113969 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.118696 (+ 4727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.120101 (+ 1405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.120106 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.120109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.120312 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.121857 (+ 1545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.121862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.121865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.122246 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.122451 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.123884 (+ 1433us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.124127 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 20:38:29.126171 31890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.078971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.23.200.1:59165)
0806 20:38:29.090797 (+ 11826us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:29.090816 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:29.090941 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:29.091629 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:29.091641 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:29.092902 (+ 1261us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:29.098243 (+ 5341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.098264 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.102438 (+ 4174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.102447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.105282 (+ 2835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.105332 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.106971 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.106979 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.107369 (+ 390us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.108155 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:29.108187 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:29.113610 (+ 5423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:29.118841 (+ 5231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.118849 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.118869 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.119969 (+ 1100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.120489 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.120496 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.120510 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.121650 (+ 1140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.122566 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:29.122576 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:29.123251 (+ 675us) client_negotiation.cc:769] Sending connection context
0806 20:38:29.123721 (+ 470us) client_negotiation.cc:240] Negotiation successful
0806 20:38:29.124384 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11677,"mutex_wait_us":783,"thread_start_us":11503,"threads_started":1}
I20260806 20:38:29.127923 31883 heartbeater.cc:347] Connected to a master server at 127.23.200.60:45777
I20260806 20:38:29.128319 31883 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:29.128551 31888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.077084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:54849)
0806 20:38:29.078259 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:29.078340 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:29.087253 (+ 8913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:29.092004 (+ 4751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:29.092008 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:29.092867 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:29.094655 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.094691 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.105506 (+ 10815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.105511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.107559 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.107576 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.108993 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.109002 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.109253 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.110187 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:29.110211 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:29.114555 (+ 4344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:29.119017 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.119023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.119028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.119914 (+ 886us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.120431 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.120437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.120444 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.121940 (+ 1496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.126977 (+ 5037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:29.126985 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:29.127215 (+ 230us) client_negotiation.cc:769] Sending connection context
0806 20:38:29.127364 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 20:38:29.127828 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":33,"spinlock_wait_cycles":1917184,"thread_start_us":1814,"threads_started":1}
I20260806 20:38:29.129518 31883 heartbeater.cc:511] Master 127.23.200.60:45777 requested a full tablet report, sending...
I20260806 20:38:29.130666 31887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.076226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:54849 (local address 127.23.200.62:41157)
0806 20:38:29.091721 (+ 15495us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.091725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.091741 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:38:29.091790 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.091793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.091851 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.091977 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.094947 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.096554 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.107794 (+ 11240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.108571 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.108702 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.114795 (+ 6093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.114801 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.114805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.114864 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.118863 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.120052 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.120057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.120061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.120240 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.122068 (+ 1828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.122076 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.122081 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.125670 (+ 3589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.128642 (+ 2972us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.128830 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.130184 (+ 1354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15394,"thread_start_us":15852,"threads_started":1}
I20260806 20:38:29.131633 31893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.095071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.23.200.1:50017)
0806 20:38:29.107934 (+ 12863us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:29.107954 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:29.108153 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:29.108842 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:29.108847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:29.109680 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:29.110659 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.110690 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.112415 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.112421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.113577 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.113589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.114977 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.114986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.115236 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.115836 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:29.115871 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:29.119389 (+ 3518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:29.125990 (+ 6601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.126013 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.126020 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.128039 (+ 2019us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.129016 (+ 977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.129044 (+ 28us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.129054 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.129314 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.129915 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:29.129920 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:29.130084 (+ 164us) client_negotiation.cc:769] Sending connection context
0806 20:38:29.130234 (+ 150us) client_negotiation.cc:240] Negotiation successful
0806 20:38:29.130558 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12675,"thread_start_us":12512,"threads_started":1}
I20260806 20:38:29.132115 31689 ts_manager.cc:194] Registered new tserver with Master: af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1:42143)
I20260806 20:38:29.132531 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.094786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:50017 (local address 127.23.200.61:33647)
0806 20:38:29.105980 (+ 11194us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.105988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.108086 (+ 2098us) server_negotiation.cc:407] Connection header received
0806 20:38:29.108314 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.108321 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.108452 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.108609 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.110926 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.112267 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.113716 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.114546 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.114673 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.119631 (+ 4958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.119637 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.119642 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.119684 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.125269 (+ 5585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.128217 (+ 2948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.128226 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.128232 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.128418 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.129460 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.129465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.129470 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.129785 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.129949 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.131219 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.132096 (+ 877us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11064,"thread_start_us":11244,"threads_started":1}
I20260806 20:38:29.132843 31884 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41157
I20260806 20:38:29.133016 31884 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:29.133728 31885 heartbeater.cc:347] Connected to a master server at 127.23.200.61:33647
I20260806 20:38:29.133769 31884 heartbeater.cc:511] Master 127.23.200.62:41157 requested a full tablet report, sending...
I20260806 20:38:29.133868 31885 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:29.134764 31885 heartbeater.cc:511] Master 127.23.200.61:33647 requested a full tablet report, sending...
I20260806 20:38:29.135555 30862 ts_manager.cc:194] Re-registered known tserver with Master: af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1:42143)
I20260806 20:38:29.136400 31486 ts_manager.cc:194] Registered new tserver with Master: af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1:42143)
I20260806 20:38:29.136734 30862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:54849
I20260806 20:38:29.212376 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 31067
I20260806 20:38:29.231796 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:33665
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:41157
--webserver_port=36999
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:38:29.395902 31646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: NON_VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: true } } }
I20260806 20:38:29.396333 31646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:29.397769 31896 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: attempting to promote NON_VOTER f49924b2bf0c4ac999c321d80fbcd397 to VOTER
I20260806 20:38:29.398397 31896 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } }
I20260806 20:38:29.400308 31700 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 LEARNER]: Refusing update from remote peer 53097dc5927343c4ab0d29474bb88790: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 20:38:29.400326 31496 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Refusing update from remote peer 53097dc5927343c4ab0d29474bb88790: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 20:38:29.400920 31897 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 20:38:29.401360 31896 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 20:38:29.404011 31646 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.406808 31496 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.409240 31646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.409555 31646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:29.409830 31646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.410116 31646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:29.414753 31900 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.415143 31900 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:38:29.416121 31897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.416399 31897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:29.416786 31700 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.420312 31901 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "53097dc5927343c4ab0d29474bb88790" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:29.420722 31901 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: This master's current role is: FOLLOWER
W20260806 20:38:29.561746 31895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:29.561993 31895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:29.562125 31895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:29.569574 31895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:29.569716 31895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:29.569877 31895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:29.579605 31895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:33665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=36999
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:29.580907 31895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:29.582209 31895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:29.595135 31907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:29.595184 31906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:29.595217 31909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:29.595988 31895 server_base.cc:1034] running on GCE node
I20260806 20:38:29.596616 31895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:29.597517 31895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:29.598734 31895 hybrid_clock.cc:648] HybridClock initialized: now 1786048709598697 us; error 50 us; skew 500 ppm
I20260806 20:38:29.599037 31895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048709, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:29.602383 31895 init.cc:377] Logged in from keytab as oryx/127.23.200.2@KRBTEST.COM (short username oryx)
I20260806 20:38:29.603849 31895 webserver.cc:460] Webserver started at http://127.23.200.2:36999/ using document root <none> and password file <none>
I20260806 20:38:29.604544 31895 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:29.604653 31895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:29.609618 31895 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:38:29.612128 31916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:29.613337 31895 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 20:38:29.613463 31895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "47c8d6d62daa401dbbb4d66e3a6377a9"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "716f7013e6523586038b7437fc12f30b"
server_key_iv: "0f76390f738b4de42f0e582c868d0b84"
server_key_version: "encryptionkey@0"
I20260806 20:38:29.614427 31895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:29.639542 31895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:29.645254 31895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:29.645578 31895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:29.646907 31895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:29.648749 31895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:29.648876 31895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:29.648986 31895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:29.649097 31895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:29.688531 31895 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33665
I20260806 20:38:29.688642 32036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33665 every 8 connection(s)
I20260806 20:38:29.690594 31895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:29.695451 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 31895
I20260806 20:38:29.695745 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 31207
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048709, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
I20260806 20:38:29.719081 32042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.693706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:48327 (local address 127.23.200.61:33647)
0806 20:38:29.694323 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.694332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.695793 (+ 1461us) server_negotiation.cc:407] Connection header received
0806 20:38:29.699682 (+ 3889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.699691 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.699787 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.700069 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.701982 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.703416 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.706378 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.707122 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.707262 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.712366 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.712374 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.712380 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.712442 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.715830 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.716569 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.716575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.716579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.716717 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.717205 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.717210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.717213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.717480 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.717637 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.718394 (+ 757us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.718608 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":352,"threads_started":1}
I20260806 20:38:29.720117 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.694521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.23.200.2:48327)
0806 20:38:29.695622 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:29.695681 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:29.699129 (+ 3448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:29.700280 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:29.700293 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:29.701052 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:29.701773 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.701792 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.703624 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.703631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.706158 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.706176 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.707597 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.707608 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.708050 (+ 442us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.708908 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:29.708945 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:29.712092 (+ 3147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:29.715991 (+ 3899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.715999 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.716014 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.716418 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.716837 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.716841 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.716845 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.717061 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.717623 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:29.717630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:29.718018 (+ 388us) client_negotiation.cc:769] Sending connection context
0806 20:38:29.718277 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 20:38:29.718820 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":688,"thread_start_us":8330,"threads_started":1}
I20260806 20:38:29.721647 32039 heartbeater.cc:347] Connected to a master server at 127.23.200.61:33647
I20260806 20:38:29.721985 32039 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:29.723094 32039 heartbeater.cc:511] Master 127.23.200.61:33647 requested a full tablet report, sending...
I20260806 20:38:29.725052 31486 ts_manager.cc:194] Registered new tserver with Master: 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665)
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048709, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048709, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.60@KRBTEST.COM
I20260806 20:38:29.747355 32047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.697866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:45469)
0806 20:38:29.717253 (+ 19387us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:29.717277 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:29.717445 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:29.723144 (+ 5699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:29.723152 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:29.724370 (+ 1218us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:29.724694 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.724709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.727558 (+ 2849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.727565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.729941 (+ 2376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.729960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.731154 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.731171 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.731338 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.732938 (+ 1600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:29.732959 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:29.736584 (+ 3625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:29.741901 (+ 5317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.741909 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.741914 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.742777 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.743529 (+ 752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.743534 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.743538 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.743798 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.745149 (+ 1351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:29.745165 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:29.745390 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 20:38:29.745543 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 20:38:29.745926 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19247,"thread_start_us":19073,"threads_started":1}
I20260806 20:38:29.747761 32043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.694001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:45469 (local address 127.23.200.62:41157)
0806 20:38:29.722547 (+ 28546us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.722555 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.722580 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:38:29.722652 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.722660 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.722761 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.722893 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.724886 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.726244 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.730088 (+ 3844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.730879 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.731032 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.736803 (+ 5771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.736808 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.736811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.736850 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.741664 (+ 4814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.742983 (+ 1319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.742992 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.742995 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.743358 (+ 363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.744121 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.744127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.744131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.744671 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.744963 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.746337 (+ 1374us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.746587 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28390,"thread_start_us":30846,"threads_started":1}
I20260806 20:38:29.748951 32038 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41157
I20260806 20:38:29.749164 32038 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:29.749795 32038 heartbeater.cc:511] Master 127.23.200.62:41157 requested a full tablet report, sending...
I20260806 20:38:29.750986 30862 ts_manager.cc:194] Re-registered known tserver with Master: 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665)
I20260806 20:38:29.751927 30862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:45469
Aug 06 20:38:29 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048704, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.3@KRBTEST.COM
I20260806 20:38:29.754110 32044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.698744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.23.200.2:52245)
0806 20:38:29.720953 (+ 22209us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:29.720971 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:29.721060 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:29.726271 (+ 5211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:29.726279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:29.726969 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:29.727378 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.727393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.730151 (+ 2758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.730158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.732617 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:29.732629 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.733874 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.733882 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:29.734359 (+ 477us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.735230 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:29.735259 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:29.739044 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:29.748861 (+ 9817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.748868 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.748872 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.749614 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.750200 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:29.750207 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:29.750214 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:29.750450 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:29.752825 (+ 2375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:29.752830 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:29.753028 (+ 198us) client_negotiation.cc:769] Sending connection context
0806 20:38:29.753256 (+ 228us) client_negotiation.cc:240] Negotiation successful
0806 20:38:29.753576 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22099,"thread_start_us":38909,"threads_started":1}
I20260806 20:38:29.755067 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.693591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:52245 (local address 127.23.200.60:45777)
0806 20:38:29.725652 (+ 32061us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.725660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.725684 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:38:29.725754 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.725761 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.725861 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.725985 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.727548 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.729095 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.732756 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.733685 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.733808 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.739318 (+ 5510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.739323 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.739327 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.739372 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.748641 (+ 9269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.749822 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.749827 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.749840 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.750046 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.752290 (+ 2244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.752295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.752298 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.752623 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.752766 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.754473 (+ 1707us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.754704 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31907,"thread_start_us":37457,"threads_started":1}
I20260806 20:38:29.756186 32037 heartbeater.cc:347] Connected to a master server at 127.23.200.60:45777
I20260806 20:38:29.756389 32037 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:29.757041 32037 heartbeater.cc:511] Master 127.23.200.60:45777 requested a full tablet report, sending...
I20260806 20:38:29.758123 31689 ts_manager.cc:194] Registered new tserver with Master: 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665)
I20260806 20:38:29.762544 32046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:29.697327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.23.200.3:37665)
0806 20:38:29.737086 (+ 39759us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:29.737098 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:29.738763 (+ 1665us) server_negotiation.cc:407] Connection header received
0806 20:38:29.738973 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:29.738982 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:29.739308 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:29.739536 (+ 228us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:29.742450 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.744859 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.747426 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:29.748698 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:29.748820 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:29.755764 (+ 6944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:29.755775 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:29.755788 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:29.755820 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:29.759523 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.760159 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.760172 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.760173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.760371 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:29.760758 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:29.760762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:29.760767 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:29.761143 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:29.761503 (+ 360us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:29.761847 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 20:38:29.762113 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39501,"thread_start_us":40115,"threads_started":1}
I20260806 20:38:29.763840 31333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:29.849437 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 31207
I20260806 20:38:29.867213 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:37665
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:41157
--webserver_port=46029
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:38:30.139027 31884 heartbeater.cc:503] Master 127.23.200.62:41157 was elected leader, sending a full tablet report...
W20260806 20:38:30.200347 32050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:30.200624 32050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:30.200753 32050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:30.208283 32050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:30.208387 32050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:30.208504 32050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:30.217857 32050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:37665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=46029
--tserver_master_addrs=127.23.200.62:41157,127.23.200.61:33647,127.23.200.60:45777
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:30.219751 32050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:30.221284 32050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:30.235008 32055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:30.235101 32056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:30.235069 32058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:30.235877 32050 server_base.cc:1034] running on GCE node
I20260806 20:38:30.236438 32050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:30.237385 32050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:30.238618 32050 hybrid_clock.cc:648] HybridClock initialized: now 1786048710238584 us; error 47 us; skew 500 ppm
I20260806 20:38:30.238919 32050 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:30 dist-test-slave-tm1g krb5kdc[30816](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048710, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:30.242043 32050 init.cc:377] Logged in from keytab as oryx/127.23.200.3@KRBTEST.COM (short username oryx)
I20260806 20:38:30.243568 32050 webserver.cc:460] Webserver started at http://127.23.200.3:46029/ using document root <none> and password file <none>
I20260806 20:38:30.244246 32050 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:30.244405 32050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:30.249913 32050 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:38:30.252542 32065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:30.254204 32050 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:38:30.254410 32050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "65856518b128475a93a8ffb622c1c1af"
format_stamp: "Formatted at 2026-08-06 20:38:26 on dist-test-slave-tm1g"
server_key: "d36382500336c22f7e3ede150c34291b"
server_key_iv: "5c3bfaae6bc8226c4352670ebf82a0d1"
server_key_version: "encryptionkey@0"
I20260806 20:38:30.255728 32050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:30.273833 32050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:30.277712 32050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:30.278003 32050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:30.278991 32050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:30.280380 32050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:30.280465 32050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:30.280609 32050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:30.280710 32050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:30.313898 32050 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37665
I20260806 20:38:30.313989 32181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37665 every 8 connection(s)
I20260806 20:38:30.315377 32050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:30.318696 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 32050
Aug 06 20:38:30 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048710, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
Aug 06 20:38:30 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048710, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.61@KRBTEST.COM
Aug 06 20:38:30 dist-test-slave-tm1g krb5kdc[30816](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048710, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.60@KRBTEST.COM
I20260806 20:38:30.384466 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.318366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:48839 (local address 127.23.200.61:33647)
0806 20:38:30.331226 (+ 12860us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.331231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.347074 (+ 15843us) server_negotiation.cc:407] Connection header received
0806 20:38:30.350126 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.350131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.350209 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.350338 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.356438 (+ 6100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.357468 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.366521 (+ 9053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.367340 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.367457 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.374344 (+ 6887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.374352 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.374356 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.374406 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.379248 (+ 4842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.380929 (+ 1681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.380935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.380939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.381105 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.381653 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.381659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.381662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.382034 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.382165 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.383660 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.383948 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12740,"thread_start_us":12624,"threads_started":1}
I20260806 20:38:30.384791 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.318633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:44903 (local address 127.23.200.62:41157)
0806 20:38:30.334729 (+ 16096us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.334739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.346392 (+ 11653us) server_negotiation.cc:407] Connection header received
0806 20:38:30.349918 (+ 3526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.349928 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.350032 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.350143 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.355786 (+ 5643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.357047 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.362724 (+ 5677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.363626 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.363789 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.372916 (+ 9127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.372922 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.372927 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.372980 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.378807 (+ 5827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.381928 (+ 3121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.381933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.381937 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.382103 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.383028 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.383034 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.383037 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.383461 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.383595 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.384083 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.384335 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15958,"thread_start_us":15862,"threads_started":1}
I20260806 20:38:30.385399 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.319347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:36865 (local address 127.23.200.60:45777)
0806 20:38:30.349311 (+ 29964us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.349318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.349340 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:38:30.352035 (+ 2695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.352045 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.352174 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.352279 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.356506 (+ 4227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.358051 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.367403 (+ 9352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.368109 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.368221 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.374414 (+ 6193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.374422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.374429 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.374498 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.379203 (+ 4705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.380813 (+ 1610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.380819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.380823 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.381007 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.381925 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.381930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.381933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.382425 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.382556 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.384301 (+ 1745us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.384830 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29827,"thread_start_us":29997,"threads_started":1}
I20260806 20:38:30.387213 32190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.319869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.23.200.3:48839)
0806 20:38:30.346937 (+ 27068us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.346962 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.350006 (+ 3044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.350726 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.350743 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.355946 (+ 5203us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:30.356289 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.356304 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.358845 (+ 2541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.358852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.366379 (+ 7527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.366393 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.367973 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.367980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.368243 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.369332 (+ 1089us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:30.369356 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:30.374113 (+ 4757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:30.379733 (+ 5620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:30.379740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:30.379744 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:30.380774 (+ 1030us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:30.381280 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:30.381286 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:30.381290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:30.381512 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:30.382263 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:30.382270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:30.382807 (+ 537us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.384700 (+ 1893us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.385281 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26894,"thread_start_us":27243,"threads_started":1}
I20260806 20:38:30.388126 32192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.318998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:44903)
0806 20:38:30.342924 (+ 23926us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.342990 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.349540 (+ 6550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.354729 (+ 5189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.354737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.355289 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:30.355628 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.355643 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.358274 (+ 2631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.358282 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.362532 (+ 4250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.362553 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.366668 (+ 4115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.366676 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.366965 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.368732 (+ 1767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:30.368765 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:30.372647 (+ 3882us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:30.379403 (+ 6756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:30.379408 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:30.379412 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:30.381740 (+ 2328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:30.382620 (+ 880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:30.382626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:30.382629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:30.382890 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:30.383584 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:30.383590 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:30.383811 (+ 221us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.383976 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.387460 (+ 3484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23609,"mutex_wait_us":908,"thread_start_us":28412,"threads_started":1}
I20260806 20:38:30.389115 32184 heartbeater.cc:347] Connected to a master server at 127.23.200.61:33647
I20260806 20:38:30.389310 32183 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41157
I20260806 20:38:30.389575 32183 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:30.389719 32188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.318998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.23.200.3:36865)
0806 20:38:30.347791 (+ 28793us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.347809 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.350113 (+ 2304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.352566 (+ 2453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.352574 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.353372 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:30.354492 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.354534 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.359623 (+ 5089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.359631 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.364914 (+ 5283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.364935 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.368588 (+ 3653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.368597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.368863 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.369530 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:30.369548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:30.373584 (+ 4036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:30.379367 (+ 5783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:30.379377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:30.379396 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:30.380610 (+ 1214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:30.381198 (+ 588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:30.381203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:30.381218 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:30.381461 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:30.383038 (+ 1577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:30.383043 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:30.383240 (+ 197us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.385665 (+ 2425us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.385981 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28647,"mutex_wait_us":1442,"thread_start_us":21995,"threads_started":1}
I20260806 20:38:30.391256 32182 heartbeater.cc:347] Connected to a master server at 127.23.200.60:45777
I20260806 20:38:30.391449 32182 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:30.389580 32184 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:30.392583 32183 heartbeater.cc:511] Master 127.23.200.62:41157 requested a full tablet report, sending...
I20260806 20:38:30.394459 30862 ts_manager.cc:194] Re-registered known tserver with Master: 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:30.395674 30862 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:44903
I20260806 20:38:30.396471 32198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.349404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.23.200.62:41157)
0806 20:38:30.365027 (+ 15623us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.365033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.367050 (+ 2017us) server_negotiation.cc:407] Connection header received
0806 20:38:30.367107 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.367112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.367209 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.367422 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.374796 (+ 7374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.378872 (+ 4076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.381436 (+ 2564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.382341 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.382484 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.386665 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.386673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.386680 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.386716 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.391517 (+ 4801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.392517 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.392523 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.392526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.392705 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.394001 (+ 1296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.394007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.394010 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.394523 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.394805 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.395880 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.396115 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15516,"thread_start_us":15398,"threads_started":1}
I20260806 20:38:30.397223 32182 heartbeater.cc:511] Master 127.23.200.60:45777 requested a full tablet report, sending...
I20260806 20:38:30.398566 31689 ts_manager.cc:194] Registered new tserver with Master: 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:30.400514 32184 heartbeater.cc:511] Master 127.23.200.61:33647 requested a full tablet report, sending...
I20260806 20:38:30.400689 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.349663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48658 (local address 127.23.200.61:33647)
0806 20:38:30.364328 (+ 14665us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.364337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.367829 (+ 3492us) server_negotiation.cc:407] Connection header received
0806 20:38:30.367893 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.367901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.367993 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.368891 (+ 898us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.372728 (+ 3837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.379495 (+ 6767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.382261 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.383826 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.386196 (+ 2370us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.391045 (+ 4849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.391054 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.391062 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.391106 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.396186 (+ 5080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.397135 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.397140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.397144 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.397319 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.397875 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.397880 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.397882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.398264 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.399731 (+ 1467us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.399822 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.400074 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14498,"thread_start_us":14370,"threads_started":1}
I20260806 20:38:30.402333 31485 ts_manager.cc:194] Registered new tserver with Master: 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:30.410256 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.365613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47904 (local address 127.23.200.60:45777)
0806 20:38:30.366833 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.366838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.393238 (+ 26400us) server_negotiation.cc:407] Connection header received
0806 20:38:30.393478 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.393483 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.393576 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.393687 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.396336 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.397678 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.400039 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.401068 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.401192 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.404503 (+ 3311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.404507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.404511 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.404544 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.407426 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.408170 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.408176 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.408179 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.408339 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.408709 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.408714 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.408717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.408976 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.409303 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.409641 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.409852 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":2004,"threads_started":1}
I20260806 20:38:30.403527 30862 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57572:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:30.412612 30862 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:30.461566 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.436732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37282 (local address 127.23.200.2:33665)
0806 20:38:30.450417 (+ 13685us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.450429 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.450464 (+ 35us) server_negotiation.cc:407] Connection header received
0806 20:38:30.450577 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.450588 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.450914 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.451153 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:30.453031 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.455349 (+ 2318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.458009 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.460155 (+ 2146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.460289 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.460547 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.460951 (+ 404us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.461128 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13428,"thread_start_us":11163,"threads_started":1}
I20260806 20:38:30.461621 32213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.436508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33665 (local address 127.0.0.1:37282)
0806 20:38:30.447450 (+ 10942us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.447492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.447674 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.451156 (+ 3482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.451165 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.451205 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:30.452780 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.452792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.455660 (+ 2868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.455669 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.457619 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.457632 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.460432 (+ 2800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.460439 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.460721 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.460765 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.460888 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.461025 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10762,"mutex_wait_us":7,"spinlock_wait_cycles":4992,"thread_start_us":894,"threads_started":1}
I20260806 20:38:30.463891 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.437202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.23.200.1:42143)
0806 20:38:30.449155 (+ 11953us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.449169 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.449209 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:38:30.449340 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.449353 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.449731 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.452656 (+ 2925us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:30.453033 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.455358 (+ 2325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.460637 (+ 5279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.462655 (+ 2018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.462789 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.463092 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.463243 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.463438 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11610,"thread_start_us":12697,"threads_started":1}
I20260806 20:38:30.464032 32214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.436065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:42143 (local address 127.0.0.1:37654)
0806 20:38:30.437804 (+ 1739us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.437830 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.438020 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.451490 (+ 13470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.451499 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.451537 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:30.452565 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.452580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.455558 (+ 2978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.455565 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.460419 (+ 4854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.460434 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.462855 (+ 2421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.462862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.463055 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.463100 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.463200 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.463327 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1572,"mutex_wait_us":1674,"thread_start_us":11716,"threads_started":1}
I20260806 20:38:30.466791 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.439413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37665 (local address 127.0.0.1:60188)
0806 20:38:30.455293 (+ 15880us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.455315 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.455485 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.457032 (+ 1547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.457037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.457061 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:30.457592 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.457604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.459847 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.459853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.462441 (+ 2588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.462453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.465626 (+ 3173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.465635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.465905 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.465987 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.466089 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.466207 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15571,"mutex_wait_us":34,"thread_start_us":16140,"threads_started":1}
I20260806 20:38:30.466214 31966 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 (DEFAULT_TABLE table=test-table [id=578f240a50734dd7a04f997506402f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:30.467402 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.440095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60188 (local address 127.23.200.3:37665)
0806 20:38:30.456534 (+ 16439us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.456539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.456565 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:38:30.456633 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.456638 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.456805 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.457044 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:30.457828 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.459670 (+ 1842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.462620 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.464609 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.464756 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.464951 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.466500 (+ 1549us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.466690 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16303,"thread_start_us":16048,"threads_started":1}
I20260806 20:38:30.467880 31966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0de2127aaf463cb4cb401b006673d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:30.468406 31812 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 (DEFAULT_TABLE table=test-table [id=578f240a50734dd7a04f997506402f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:30.469646 31812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0de2127aaf463cb4cb401b006673d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:30.474491 32114 tablet_service.cc:1467] Processing CreateTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 (DEFAULT_TABLE table=test-table [id=578f240a50734dd7a04f997506402f95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:30.476272 32114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b0de2127aaf463cb4cb401b006673d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:30.483382 32219 tablet_bootstrap.cc:492] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Bootstrap starting.
I20260806 20:38:30.485529 32220 tablet_bootstrap.cc:492] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Bootstrap starting.
I20260806 20:38:30.489516 32219 tablet_bootstrap.cc:654] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:30.489802 32220 tablet_bootstrap.cc:654] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:30.491248 32219 log.cc:824] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:30.491317 32220 log.cc:824] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:30.492129 32221 tablet_bootstrap.cc:492] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Bootstrap starting.
I20260806 20:38:30.495551 32220 tablet_bootstrap.cc:492] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: No bootstrap required, opened a new log
I20260806 20:38:30.495882 32220 ts_tablet_manager.cc:1397] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260806 20:38:30.496640 32221 tablet_bootstrap.cc:654] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:30.497066 32219 tablet_bootstrap.cc:492] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: No bootstrap required, opened a new log
I20260806 20:38:30.497339 32219 ts_tablet_manager.cc:1397] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260806 20:38:30.497869 32221 log.cc:824] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:30.507071 32221 tablet_bootstrap.cc:492] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: No bootstrap required, opened a new log
I20260806 20:38:30.507395 32221 ts_tablet_manager.cc:1397] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260806 20:38:30.509341 32220 raft_consensus.cc:348] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.509902 32220 raft_consensus.cc:374] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:30.510031 32220 raft_consensus.cc:729] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c8d6d62daa401dbbb4d66e3a6377a9, State: Initialized, Role: FOLLOWER
I20260806 20:38:30.510691 32220 consensus_queue.cc:260] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.510691 32219 raft_consensus.cc:348] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.511226 32219 raft_consensus.cc:374] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:30.511380 32219 raft_consensus.cc:729] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af44181b22d84f7ab91d75ce7b6a6e0b, State: Initialized, Role: FOLLOWER
I20260806 20:38:30.512152 32038 heartbeater.cc:503] Master 127.23.200.62:41157 was elected leader, sending a full tablet report...
I20260806 20:38:30.512058 32219 consensus_queue.cc:260] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.512924 32220 ts_tablet_manager.cc:1428] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Time spent starting tablet: real 0.017s user 0.017s sys 0.000s
I20260806 20:38:30.514781 32219 ts_tablet_manager.cc:1428] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Time spent starting tablet: real 0.017s user 0.018s sys 0.001s
I20260806 20:38:30.521824 32221 raft_consensus.cc:348] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.522352 32221 raft_consensus.cc:374] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:30.522516 32221 raft_consensus.cc:729] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65856518b128475a93a8ffb622c1c1af, State: Initialized, Role: FOLLOWER
I20260806 20:38:30.523068 32221 consensus_queue.cc:260] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.524374 32183 heartbeater.cc:503] Master 127.23.200.62:41157 was elected leader, sending a full tablet report...
I20260806 20:38:30.524775 32221 ts_tablet_manager.cc:1428] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Time spent starting tablet: real 0.017s user 0.005s sys 0.009s
I20260806 20:38:30.536648 32226 raft_consensus.cc:482] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:30.537041 32226 raft_consensus.cc:504] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.538631 32226 leader_election.cc:288] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665), 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:30.546396 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.538929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:54947 (local address 127.23.200.2:33665)
0806 20:38:30.539214 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.539223 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.540065 (+ 842us) server_negotiation.cc:407] Connection header received
0806 20:38:30.540322 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.540326 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.540403 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.540489 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:30.541182 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.541924 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.544324 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.545244 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.545450 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.545610 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.545913 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.546054 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:38:30.546463 32228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.538825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33665 (local address 127.23.200.1:54947)
0806 20:38:30.539930 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.539948 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.540145 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.540527 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.540532 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.540558 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:30.541035 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.541045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.542065 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.542070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.544147 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.544159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.545512 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.545519 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.545717 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.545757 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.545851 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.545938 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":41,"thread_start_us":795,"threads_started":1}
I20260806 20:38:30.546551 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.539012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:52645 (local address 127.23.200.3:37665)
0806 20:38:30.539242 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.539250 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.540140 (+ 890us) server_negotiation.cc:407] Connection header received
0806 20:38:30.540280 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.540285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.540344 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.540405 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:30.541207 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.541920 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.544326 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.545407 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.545616 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.545693 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.546106 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.546246 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:38:30.546831 32229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.538840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37665 (local address 127.23.200.1:52645)
0806 20:38:30.540050 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:30.540064 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:30.540163 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:30.540447 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:30.540454 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:30.540481 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:30.541039 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.541048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.542072 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.542076 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.544153 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:30.544165 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.545761 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.545766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:30.545930 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.545962 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 20:38:30.546061 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 20:38:30.546174 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":51,"thread_start_us":781,"threads_started":1}
I20260806 20:38:30.547550 31987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0de2127aaf463cb4cb401b006673d4" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" is_pre_election: true
I20260806 20:38:30.547971 32134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0de2127aaf463cb4cb401b006673d4" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65856518b128475a93a8ffb622c1c1af" is_pre_election: true
I20260806 20:38:30.548130 31987 raft_consensus.cc:2454] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 0.
I20260806 20:38:30.548453 32134 raft_consensus.cc:2454] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 0.
I20260806 20:38:30.548851 31763 leader_election.cc:302] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c8d6d62daa401dbbb4d66e3a6377a9, af44181b22d84f7ab91d75ce7b6a6e0b; no voters:
I20260806 20:38:30.549202 32226 raft_consensus.cc:2781] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:30.549388 32226 raft_consensus.cc:482] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:30.549535 32226 raft_consensus.cc:3037] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:30.552275 32226 raft_consensus.cc:504] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.552878 32226 leader_election.cc:288] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 election: Requested vote from peers 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665), 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:30.553269 32134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0de2127aaf463cb4cb401b006673d4" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65856518b128475a93a8ffb622c1c1af"
I20260806 20:38:30.553272 31987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b0de2127aaf463cb4cb401b006673d4" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9"
I20260806 20:38:30.553472 31987 raft_consensus.cc:3037] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:30.553508 32134 raft_consensus.cc:3037] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:30.556140 32134 raft_consensus.cc:2454] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 1.
I20260806 20:38:30.556272 31987 raft_consensus.cc:2454] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 1.
I20260806 20:38:30.556741 31766 leader_election.cc:302] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 65856518b128475a93a8ffb622c1c1af, af44181b22d84f7ab91d75ce7b6a6e0b; no voters:
I20260806 20:38:30.557096 32226 raft_consensus.cc:2781] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:30.557826 32226 raft_consensus.cc:686] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 LEADER]: Becoming Leader. State: Replica: af44181b22d84f7ab91d75ce7b6a6e0b, State: Running, Role: LEADER
I20260806 20:38:30.558691 32226 consensus_queue.cc:237] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:30.563969 30861 catalog_manager.cc:5495] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1). New cstate: current_term: 1 leader_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } health_report { overall_health: UNKNOWN } } }
W20260806 20:38:30.568363 32185 tablet.cc:2369] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:38:30.602072 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.595539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37668 (local address 127.23.200.1:42143)
0806 20:38:30.595740 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.595747 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.595764 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:38:30.595999 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.596003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.596081 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.596231 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:30.596706 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.598012 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.599703 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.600177 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.600321 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.600853 (+ 532us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:30.601360 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:30.601571 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.601623 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.601700 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 20:38:30.602356 31891 tablet.cc:2369] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:38:30.611670 31987 raft_consensus.cc:1261] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Refusing update from remote peer af44181b22d84f7ab91d75ce7b6a6e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:30.611644 32134 raft_consensus.cc:1261] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Refusing update from remote peer af44181b22d84f7ab91d75ce7b6a6e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:30.612699 32230 consensus_queue.cc:1035] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:30.613168 32226 consensus_queue.cc:1035] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:30.627913 32234 mvcc.cc:205] Tried to move back new op lower bound from 7315655518653784064 to 7315655518452740096. Current Snapshot: MvccSnapshot[applied={T|T < 7315655518653784064}]
I20260806 20:38:30.648131 30861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:30.648424 30861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57572:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:30.656917 30861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53097dc5927343c4ab0d29474bb88790: Sending DeleteTablet for 3 replicas of tablet 8b0de2127aaf463cb4cb401b006673d4
I20260806 20:38:30.657871 32114 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:30 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:60188
I20260806 20:38:30.657979 32113 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:30 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:60188
I20260806 20:38:30.659030 31812 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:30 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:37654
I20260806 20:38:30.659382 31966 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b0de2127aaf463cb4cb401b006673d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:30 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:37282
I20260806 20:38:30.660537 32246 tablet_replica.cc:330] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: stopping tablet replica
I20260806 20:38:30.661245 32246 raft_consensus.cc:2229] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:30.661926 32246 raft_consensus.cc:2258] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:30.664556 32246 ts_tablet_manager.cc:1905] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:30.665499 32244 tablet_replica.cc:330] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: stopping tablet replica
I20260806 20:38:30.665880 32244 raft_consensus.cc:2229] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:30.667733 32244 raft_consensus.cc:2258] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:30.671849 32244 ts_tablet_manager.cc:1905] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:30.673167 32246 ts_tablet_manager.cc:1918] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:30.673409 32246 log.cc:1196] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/8b0de2127aaf463cb4cb401b006673d4
I20260806 20:38:30.674084 32246 ts_tablet_manager.cc:1939] T 8b0de2127aaf463cb4cb401b006673d4 P 47c8d6d62daa401dbbb4d66e3a6377a9: Deleting consensus metadata
I20260806 20:38:30.675113 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 30829
I20260806 20:38:30.676436 30893 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:30.676350 30845 catalog_manager.cc:4841] TS 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665): tablet 8b0de2127aaf463cb4cb401b006673d4 (table test-table [id=578f240a50734dd7a04f997506402f95]) successfully deleted
I20260806 20:38:30.682463 32245 tablet_replica.cc:330] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: stopping tablet replica
I20260806 20:38:30.682926 32245 raft_consensus.cc:2229] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:30.683421 32244 ts_tablet_manager.cc:1918] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:30.683596 32244 log.cc:1196] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/8b0de2127aaf463cb4cb401b006673d4
I20260806 20:38:30.683821 32245 raft_consensus.cc:2258] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:30.684219 32244 ts_tablet_manager.cc:1939] T 8b0de2127aaf463cb4cb401b006673d4 P 65856518b128475a93a8ffb622c1c1af: Deleting consensus metadata
I20260806 20:38:30.686182 32245 ts_tablet_manager.cc:1905] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:30.695117 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.663103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48662 (local address 127.23.200.61:33647)
0806 20:38:30.663291 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.663297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.673412 (+ 10115us) server_negotiation.cc:407] Connection header received
0806 20:38:30.676431 (+ 3019us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.676437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.676524 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.676701 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.679752 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.680522 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.681917 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.682719 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.682859 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.687214 (+ 4355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.687221 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.687226 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.687271 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.691074 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.692145 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.692151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.692154 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.692309 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.692962 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.692967 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.692979 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.693468 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.693773 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.694501 (+ 728us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.694746 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 20:38:30.715274 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.673865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47914 (local address 127.23.200.60:45777)
0806 20:38:30.674057 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.674063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.697780 (+ 23717us) server_negotiation.cc:407] Connection header received
0806 20:38:30.698067 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.698073 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.698164 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.698239 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.700411 (+ 2172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.701605 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.703583 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.704300 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.704401 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.708194 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.708200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.708205 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.708247 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.711918 (+ 3671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.712787 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.712792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.712796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.712956 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.713417 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.713422 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.713426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.713767 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.714029 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.714680 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.714924 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:38:30.717166 32245 ts_tablet_manager.cc:1918] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:30.717401 32245 log.cc:1196] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/8b0de2127aaf463cb4cb401b006673d4
I20260806 20:38:30.718144 32245 ts_tablet_manager.cc:1939] T 8b0de2127aaf463cb4cb401b006673d4 P af44181b22d84f7ab91d75ce7b6a6e0b: Deleting consensus metadata
I20260806 20:38:30.819746 30848 catalog_manager.cc:4841] TS 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665): tablet 8b0de2127aaf463cb4cb401b006673d4 (table test-table [id=578f240a50734dd7a04f997506402f95]) successfully deleted
I20260806 20:38:30.822970 30846 catalog_manager.cc:4841] TS af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1:42143): tablet 8b0de2127aaf463cb4cb401b006673d4 (table test-table [id=578f240a50734dd7a04f997506402f95]) successfully deleted
W20260806 20:38:30.823076 30848 catalog_manager.cc:4808] TS 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665): delete failed for tablet 8b0de2127aaf463cb4cb401b006673d4 because the tablet was not found. No further retry: Not found: Tablet not found: 8b0de2127aaf463cb4cb401b006673d4
I20260806 20:38:30.823906 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 30829
I20260806 20:38:30.884936 32199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.853120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48670 (local address 127.23.200.61:33647)
0806 20:38:30.853353 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.853362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.866545 (+ 13183us) server_negotiation.cc:407] Connection header received
0806 20:38:30.867215 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.867225 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.867342 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.867436 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.872054 (+ 4618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.873247 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.874579 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.875032 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.875173 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.878444 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.878449 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.878452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.878485 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.881803 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.882615 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.882620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.882625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.882791 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.883206 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.883213 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.883216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.883618 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.883878 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.884224 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.884463 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 20:38:30.887153 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:30.853095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.23.200.60:45777)
0806 20:38:30.853308 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:30.853314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:30.870282 (+ 16968us) server_negotiation.cc:407] Connection header received
0806 20:38:30.870720 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:30.870729 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:30.870834 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:30.870926 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:30.873287 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.874468 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.876341 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:30.877067 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:30.877215 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:30.880331 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:30.880337 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:30.880342 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:30.880387 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:30.883916 (+ 3529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.884753 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.884758 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.884760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.884909 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:30.885399 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:30.885404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:30.885407 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:30.885765 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:30.886037 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:30.886638 (+ 601us) server_negotiation.cc:299] Negotiation successful
0806 20:38:30.886816 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 20:38:31.637050 32256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:31.636379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:37163)
0806 20:38:31.636774 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:31.636888 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":197,"threads_started":1}
W20260806 20:38:31.637483 32038 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:41157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:31.644063 32257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:31.643248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:36363)
0806 20:38:31.643739 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:31.643885 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":227,"threads_started":1}
W20260806 20:38:31.644423 31884 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:41157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:31.660619 32258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:31.659801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:59895)
0806 20:38:31.660251 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:31.660361 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":243,"threads_started":1}
W20260806 20:38:31.660985 32183 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:41157 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:32.155751 32259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53097dc5927343c4ab0d29474bb88790)
I20260806 20:38:32.155948 32259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } }
I20260806 20:38:32.156694 32259 leader_election.cc:288] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157), f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60:45777)
I20260806 20:38:32.169400 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.156980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:57950)
0806 20:38:32.168867 (+ 11887us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.169109 (+ 242us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11722,"thread_start_us":11887,"threads_started":1}
I20260806 20:38:32.169827 32260 raft_consensus.cc:482] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 53097dc5927343c4ab0d29474bb88790)
W20260806 20:38:32.169802 31469 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:32.170104 32260 raft_consensus.cc:504] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } }
I20260806 20:38:32.171891 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.171481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:57980)
0806 20:38:32.171673 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.171746 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260806 20:38:32.172147 31469 leader_election.cc:334] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157): Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:32.172626 32260 leader_election.cc:288] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157), f64bc7e877464bd0af21503023e019c9 (127.23.200.61:33647)
I20260806 20:38:32.174108 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.171613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:57990)
0806 20:38:32.173587 (+ 1974us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.173804 (+ 217us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1859,"thread_start_us":2085,"threads_started":1}
W20260806 20:38:32.174475 31672 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:32.176379 32265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.175959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:58004)
0806 20:38:32.176150 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.176222 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260806 20:38:32.176748 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.157026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:45777 (local address 127.0.0.1:47944)
0806 20:38:32.169240 (+ 12214us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.169258 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:32.169392 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:32.170015 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:32.170020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:32.170046 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:32.170475 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.170494 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.172138 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.172145 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.174336 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.174347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.175973 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.175979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.176165 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.176198 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 20:38:32.176280 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 20:38:32.176397 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12099,"thread_start_us":11495,"threads_started":1}
W20260806 20:38:32.176671 31672 leader_election.cc:334] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157): Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:32.177139 32263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.157407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47944 (local address 127.23.200.60:45777)
0806 20:38:32.169464 (+ 12057us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.169470 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.169492 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:38:32.169575 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.169582 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.169699 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.169823 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:32.170741 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.171956 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.174477 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.175821 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.175956 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.176059 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.176351 (+ 292us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.176498 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11914,"thread_start_us":11755,"threads_started":1}
I20260806 20:38:32.177776 31700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f64bc7e877464bd0af21503023e019c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f49924b2bf0c4ac999c321d80fbcd397" is_pre_election: true
I20260806 20:38:32.178099 31700 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f64bc7e877464bd0af21503023e019c9 in term 1.
I20260806 20:38:32.178547 31472 leader_election.cc:302] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f49924b2bf0c4ac999c321d80fbcd397, f64bc7e877464bd0af21503023e019c9; no voters: 53097dc5927343c4ab0d29474bb88790
I20260806 20:38:32.178867 32259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 20:38:32.179010 32259 raft_consensus.cc:482] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 53097dc5927343c4ab0d29474bb88790)
I20260806 20:38:32.179108 32259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:38:32.180692 32267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.171962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48684 (local address 127.23.200.61:33647)
0806 20:38:32.173198 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.173204 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.174086 (+ 882us) server_negotiation.cc:407] Connection header received
0806 20:38:32.174339 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.174344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.174417 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.174510 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:32.175326 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.176487 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.178944 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.179817 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.179912 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.180000 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.180339 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.180429 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1121,"thread_start_us":942,"threads_started":1}
I20260806 20:38:32.180728 32266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.171627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:33647 (local address 127.0.0.1:48684)
0806 20:38:32.173989 (+ 2362us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.174003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:32.174144 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:32.174572 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:32.174578 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:32.174604 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:32.175078 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.175096 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.176691 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.176696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.178730 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.178758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.179990 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.179998 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.180174 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.180209 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:38:32.180300 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 20:38:32.180422 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"thread_start_us":1720,"threads_started":1}
I20260806 20:38:32.181324 31496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f49924b2bf0c4ac999c321d80fbcd397" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f64bc7e877464bd0af21503023e019c9" is_pre_election: true
I20260806 20:38:32.182090 32259 raft_consensus.cc:504] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } }
I20260806 20:38:32.182579 31496 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f49924b2bf0c4ac999c321d80fbcd397 in current term 2: Already voted for candidate f64bc7e877464bd0af21503023e019c9 in this term.
I20260806 20:38:32.182752 32259 leader_election.cc:288] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 election: Requested vote from peers 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157), f49924b2bf0c4ac999c321d80fbcd397 (127.23.200.60:45777)
I20260806 20:38:32.183166 31700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f64bc7e877464bd0af21503023e019c9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f49924b2bf0c4ac999c321d80fbcd397"
I20260806 20:38:32.183350 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.182865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:58016)
0806 20:38:32.183111 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.183208 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260806 20:38:32.183372 31700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:38:32.183646 31674 leader_election.cc:302] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f49924b2bf0c4ac999c321d80fbcd397; no voters: 53097dc5927343c4ab0d29474bb88790, f64bc7e877464bd0af21503023e019c9
I20260806 20:38:32.184278 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.183968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:58020)
0806 20:38:32.184098 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.184179 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260806 20:38:32.184468 31469 leader_election.cc:334] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157): Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
I20260806 20:38:32.185750 31700 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f64bc7e877464bd0af21503023e019c9 in term 2.
I20260806 20:38:32.185969 32260 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:38:32.186147 31472 leader_election.cc:302] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f49924b2bf0c4ac999c321d80fbcd397, f64bc7e877464bd0af21503023e019c9; no voters: 53097dc5927343c4ab0d29474bb88790
I20260806 20:38:32.186444 32259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:38:32.186939 32259 raft_consensus.cc:686] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [term 2 LEADER]: Becoming Leader. State: Replica: f64bc7e877464bd0af21503023e019c9, State: Running, Role: LEADER
I20260806 20:38:32.187318 32259 consensus_queue.cc:237] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } }
I20260806 20:38:32.189030 32269 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f64bc7e877464bd0af21503023e019c9. Latest consensus state: current_term: 2 leader_uuid: "f64bc7e877464bd0af21503023e019c9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:32.189334 32269 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:32.189893 32271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:32.193372 32271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:32.194180 32271 catalog_manager.cc:1259] Loaded cluster ID: 312d840adb0e4e588bbb6563f1aa42ea
I20260806 20:38:32.194278 32271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:32.195490 32271 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:32.196303 32271 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Loaded TSK: 0
I20260806 20:38:32.197105 32271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:38:32.225392 31486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:32.232252 31486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:32.236333 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.235949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:58042)
0806 20:38:32.236116 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.236200 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 20:38:32.236371 31700 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [term 2 FOLLOWER]: Refusing update from remote peer f64bc7e877464bd0af21503023e019c9: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 20:38:32.237169 32269 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260806 20:38:32.237419 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.236958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.0.0.1:58046)
0806 20:38:32.237174 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.237268 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 20:38:32.237608 31469 consensus_peers.cc:487] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 -> Peer 53097dc5927343c4ab0d29474bb88790 (127.23.200.62:41157): Couldn't send request to peer 53097dc5927343c4ab0d29474bb88790. Status: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 20:38:32.239894 32260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f64bc7e877464bd0af21503023e019c9. Latest consensus state: current_term: 2 leader_uuid: "f64bc7e877464bd0af21503023e019c9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:32.240216 32260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:38:32.241565 32259 sys_catalog.cc:455] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f64bc7e877464bd0af21503023e019c9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:32.241827 32259 sys_catalog.cc:458] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:32.242380 32260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f64bc7e877464bd0af21503023e019c9" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "53097dc5927343c4ab0d29474bb88790" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41157 } } peers { permanent_uuid: "f64bc7e877464bd0af21503023e019c9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 33647 } attrs { promote: false } } peers { permanent_uuid: "f49924b2bf0c4ac999c321d80fbcd397" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 45777 } attrs { promote: false } } }
I20260806 20:38:32.242622 32260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f49924b2bf0c4ac999c321d80fbcd397 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:38:32.256989 32261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.249346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:42143 (local address 127.0.0.1:37682)
0806 20:38:32.249663 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.249683 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:32.249785 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:32.251356 (+ 1571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:32.251364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:32.251390 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:32.251592 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.251600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.253087 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.253092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.254499 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.254513 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.256263 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.256269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.256461 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.256499 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:38:32.256565 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 20:38:32.256694 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260806 20:38:32.257923 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.249465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.23.200.1:42143)
0806 20:38:32.250998 (+ 1533us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.251007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.251030 (+ 23us) server_negotiation.cc:407] Connection header received
0806 20:38:32.251109 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.251115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.251195 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.251805 (+ 610us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:32.251997 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.252941 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.254688 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.256071 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.257319 (+ 1248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.257420 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.257487 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.257638 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":1852,"threads_started":1}
I20260806 20:38:32.258481 32262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.249782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33665 (local address 127.0.0.1:37294)
0806 20:38:32.249952 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.249976 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:32.250128 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:32.251044 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:32.251050 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:32.251079 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:32.251673 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.251678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.254014 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.254020 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.256122 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.256139 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.257725 (+ 1586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.257731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.257908 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.257940 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 20:38:32.258041 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 20:38:32.258179 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":259}
I20260806 20:38:32.258680 31812 tablet_service.cc:1467] Processing CreateTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 (DEFAULT_TABLE table=test-table [id=053e176d3ad9400c99f82ea9ed4ba53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:32.258831 32279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.250133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37294 (local address 127.23.200.2:33665)
0806 20:38:32.250667 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.250673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.250691 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:38:32.250764 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.250770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.250862 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.251445 (+ 583us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:32.251827 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.253851 (+ 2024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.256303 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.257496 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.257615 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.257700 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.258414 (+ 714us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.258552 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
I20260806 20:38:32.259156 31812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c3ceea1abd414898cf7aa58a030f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:32.259459 31966 tablet_service.cc:1467] Processing CreateTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 (DEFAULT_TABLE table=test-table [id=053e176d3ad9400c99f82ea9ed4ba53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:32.259917 31966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c3ceea1abd414898cf7aa58a030f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:32.261482 32280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.250721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37665 (local address 127.0.0.1:60192)
0806 20:38:32.251268 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.251286 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:32.251383 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:32.254965 (+ 3582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:32.254970 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:32.254995 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:32.255380 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.255388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.256734 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.256743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.258853 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:32.258867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.260739 (+ 1872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.260747 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:32.260915 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.260948 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 20:38:32.261035 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 20:38:32.261169 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":508,"threads_started":1}
I20260806 20:38:32.262260 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.251092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60192 (local address 127.23.200.3:37665)
0806 20:38:32.254704 (+ 3612us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.254708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.254723 (+ 15us) server_negotiation.cc:407] Connection header received
0806 20:38:32.254779 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.254783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.254835 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.254950 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:32.255553 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.256609 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.259028 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.260338 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.260461 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.260593 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.261788 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.261941 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3502,"thread_start_us":3410,"threads_started":1}
I20260806 20:38:32.262943 32114 tablet_service.cc:1467] Processing CreateTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 (DEFAULT_TABLE table=test-table [id=053e176d3ad9400c99f82ea9ed4ba53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:32.263279 32114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02c3ceea1abd414898cf7aa58a030f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:32.266723 32282 tablet_bootstrap.cc:492] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Bootstrap starting.
I20260806 20:38:32.268440 32282 tablet_bootstrap.cc:654] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:32.268571 32284 tablet_bootstrap.cc:492] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Bootstrap starting.
I20260806 20:38:32.269801 32283 tablet_bootstrap.cc:492] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Bootstrap starting.
I20260806 20:38:32.270584 32284 tablet_bootstrap.cc:654] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:32.271548 32283 tablet_bootstrap.cc:654] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:32.271839 32282 tablet_bootstrap.cc:492] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: No bootstrap required, opened a new log
I20260806 20:38:32.272033 32282 ts_tablet_manager.cc:1397] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 20:38:32.272840 32282 raft_consensus.cc:348] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.273041 32284 tablet_bootstrap.cc:492] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: No bootstrap required, opened a new log
I20260806 20:38:32.273113 32282 raft_consensus.cc:374] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:32.273170 32284 ts_tablet_manager.cc:1397] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260806 20:38:32.273204 32282 raft_consensus.cc:729] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c8d6d62daa401dbbb4d66e3a6377a9, State: Initialized, Role: FOLLOWER
I20260806 20:38:32.273495 32282 consensus_queue.cc:260] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.273985 32284 raft_consensus.cc:348] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.274211 32284 raft_consensus.cc:374] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:32.274356 32282 ts_tablet_manager.cc:1428] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 20:38:32.274353 32284 raft_consensus.cc:729] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65856518b128475a93a8ffb622c1c1af, State: Initialized, Role: FOLLOWER
I20260806 20:38:32.274695 32283 tablet_bootstrap.cc:492] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: No bootstrap required, opened a new log
I20260806 20:38:32.274619 32284 consensus_queue.cc:260] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.274904 32283 ts_tablet_manager.cc:1397] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 20:38:32.275918 32283 raft_consensus.cc:348] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.276290 32283 raft_consensus.cc:374] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:32.276454 32283 raft_consensus.cc:729] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af44181b22d84f7ab91d75ce7b6a6e0b, State: Initialized, Role: FOLLOWER
I20260806 20:38:32.276837 32283 consensus_queue.cc:260] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.278651 32290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.275208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:43937)
0806 20:38:32.278148 (+ 2940us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.278355 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2398,"thread_start_us":2257,"threads_started":1}
I20260806 20:38:32.281113 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.279963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:56571)
0806 20:38:32.280610 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.280821 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":307,"threads_started":1}
I20260806 20:38:32.281244 32039 heartbeater.cc:503] Master 127.23.200.61:33647 was elected leader, sending a full tablet report...
I20260806 20:38:32.281432 32284 ts_tablet_manager.cc:1428] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Time spent starting tablet: real 0.008s user 0.001s sys 0.008s
I20260806 20:38:32.283246 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.282366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:56327)
0806 20:38:32.282822 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.282988 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":639,"threads_started":1}
I20260806 20:38:32.283707 32184 heartbeater.cc:503] Master 127.23.200.61:33647 was elected leader, sending a full tablet report...
I20260806 20:38:32.285384 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.284883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:57305)
0806 20:38:32.285183 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.285271 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 20:38:32.286288 32283 ts_tablet_manager.cc:1428] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20260806 20:38:32.287989 31885 heartbeater.cc:503] Master 127.23.200.61:33647 was elected leader, sending a full tablet report...
I20260806 20:38:32.288708 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.286347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:56461)
0806 20:38:32.287920 (+ 1573us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.288109 (+ 189us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1444,"thread_start_us":1314,"threads_started":1}
I20260806 20:38:32.289913 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.289546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:33259)
0806 20:38:32.289703 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.289781 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260806 20:38:32.290534 32291 raft_consensus.cc:482] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:32.290716 32291 raft_consensus.cc:504] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.291333 32291 leader_election.cc:288] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665), 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:32.291785 32134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c3ceea1abd414898cf7aa58a030f84" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65856518b128475a93a8ffb622c1c1af" is_pre_election: true
I20260806 20:38:32.291785 31987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c3ceea1abd414898cf7aa58a030f84" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" is_pre_election: true
I20260806 20:38:32.292006 31987 raft_consensus.cc:2454] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 0.
I20260806 20:38:32.291991 32134 raft_consensus.cc:2454] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 0.
I20260806 20:38:32.292449 31766 leader_election.cc:302] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 47c8d6d62daa401dbbb4d66e3a6377a9, 65856518b128475a93a8ffb622c1c1af, af44181b22d84f7ab91d75ce7b6a6e0b; no voters:
I20260806 20:38:32.292737 32291 raft_consensus.cc:2781] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:32.292858 32291 raft_consensus.cc:482] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:32.292948 32291 raft_consensus.cc:3037] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:32.294997 32291 raft_consensus.cc:504] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.295549 32291 leader_election.cc:288] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 election: Requested vote from peers 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665), 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665)
I20260806 20:38:32.295897 31987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c3ceea1abd414898cf7aa58a030f84" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9"
I20260806 20:38:32.295924 32134 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02c3ceea1abd414898cf7aa58a030f84" candidate_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65856518b128475a93a8ffb622c1c1af"
I20260806 20:38:32.296099 31987 raft_consensus.cc:3037] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:32.296104 32134 raft_consensus.cc:3037] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:32.298485 31987 raft_consensus.cc:2454] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 1.
I20260806 20:38:32.298529 32134 raft_consensus.cc:2454] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af44181b22d84f7ab91d75ce7b6a6e0b in term 1.
I20260806 20:38:32.298909 31763 leader_election.cc:302] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c8d6d62daa401dbbb4d66e3a6377a9, af44181b22d84f7ab91d75ce7b6a6e0b; no voters:
I20260806 20:38:32.299189 32291 raft_consensus.cc:2781] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:32.299575 32291 raft_consensus.cc:686] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 LEADER]: Becoming Leader. State: Replica: af44181b22d84f7ab91d75ce7b6a6e0b, State: Running, Role: LEADER
I20260806 20:38:32.299873 32291 consensus_queue.cc:237] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } }
I20260806 20:38:32.300577 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.300078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:56555)
0806 20:38:32.300268 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.300420 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260806 20:38:32.302512 31486 catalog_manager.cc:5495] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b reported cstate change: term changed from 0 to 1, leader changed from <none> to af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1). New cstate: current_term: 1 leader_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af44181b22d84f7ab91d75ce7b6a6e0b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42143 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:32.318692 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.313759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.23.200.1:42143)
0806 20:38:32.313969 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.313975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.313994 (+ 19us) server_negotiation.cc:407] Connection header received
0806 20:38:32.314081 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.314086 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.314176 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.314279 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:32.314649 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.315679 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.316892 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.317531 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.317644 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.317999 (+ 355us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:32.318119 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:32.318203 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.318365 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.318442 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 20:38:32.321539 32134 raft_consensus.cc:1261] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Refusing update from remote peer af44181b22d84f7ab91d75ce7b6a6e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:32.321832 31987 raft_consensus.cc:1261] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Refusing update from remote peer af44181b22d84f7ab91d75ce7b6a6e0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:32.322099 32291 consensus_queue.cc:1035] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "65856518b128475a93a8ffb622c1c1af" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:32.322571 32295 consensus_queue.cc:1035] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47c8d6d62daa401dbbb4d66e3a6377a9" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:32.324980 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.324422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:33889)
0806 20:38:32.324663 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.324784 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 20:38:32.325613 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.325188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:33089)
0806 20:38:32.325365 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.325458 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 20:38:32.332250 32293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.331643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.3:47107)
0806 20:38:32.331925 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.332045 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260806 20:38:32.340715 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.340261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:44665)
0806 20:38:32.340476 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.340580 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 20:38:32.345134 31486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:32.345400 31486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48670:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:32.345568 32292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.345113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.2:47783)
0806 20:38:32.345276 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.345402 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 20:38:32.349233 32294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.348884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41157 (local address 127.23.200.1:45975)
0806 20:38:32.349026 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:32.349107 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:41157: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 20:38:32.351967 31486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f64bc7e877464bd0af21503023e019c9: Sending DeleteTablet for 3 replicas of tablet 02c3ceea1abd414898cf7aa58a030f84
I20260806 20:38:32.352936 31812 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:32 UTC) from {username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:37682
I20260806 20:38:32.353119 31966 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:32 UTC) from {username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:37294
I20260806 20:38:32.353618 31965 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:32 UTC) from {username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:37294
I20260806 20:38:32.353754 32312 tablet_replica.cc:330] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: stopping tablet replica
I20260806 20:38:32.353947 32312 raft_consensus.cc:2229] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:32.354261 32312 raft_consensus.cc:2258] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:32.354375 32114 tablet_service.cc:1514] Processing DeleteTablet for tablet 02c3ceea1abd414898cf7aa58a030f84 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:32 UTC) from {username='oryx', principal='oryx/127.23.200.61@KRBTEST.COM'} at 127.0.0.1:60192
I20260806 20:38:32.355010 32313 tablet_replica.cc:330] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: stopping tablet replica
I20260806 20:38:32.355197 32313 raft_consensus.cc:2229] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:32.355399 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 31673
I20260806 20:38:32.355505 32313 raft_consensus.cc:2258] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:32.356820 32313 ts_tablet_manager.cc:1905] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:32.357746 32312 ts_tablet_manager.cc:1905] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:32.358613 32310 tablet_replica.cc:330] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: stopping tablet replica
I20260806 20:38:32.358788 32310 raft_consensus.cc:2229] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:32.360539 32310 raft_consensus.cc:2258] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:32.361531 32310 ts_tablet_manager.cc:1905] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:32.362978 32313 ts_tablet_manager.cc:1918] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:32.363116 32313 log.cc:1196] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/02c3ceea1abd414898cf7aa58a030f84
I20260806 20:38:32.363547 32313 ts_tablet_manager.cc:1939] T 02c3ceea1abd414898cf7aa58a030f84 P 65856518b128475a93a8ffb622c1c1af: Deleting consensus metadata
I20260806 20:38:32.364122 32312 ts_tablet_manager.cc:1918] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:32.364259 32312 log.cc:1196] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/02c3ceea1abd414898cf7aa58a030f84
I20260806 20:38:32.364579 31472 catalog_manager.cc:4841] TS 65856518b128475a93a8ffb622c1c1af (127.23.200.3:37665): tablet 02c3ceea1abd414898cf7aa58a030f84 (table test-table [id=053e176d3ad9400c99f82ea9ed4ba53c]) successfully deleted
I20260806 20:38:32.365139 32312 ts_tablet_manager.cc:1939] T 02c3ceea1abd414898cf7aa58a030f84 P 47c8d6d62daa401dbbb4d66e3a6377a9: Deleting consensus metadata
I20260806 20:38:32.366273 31469 catalog_manager.cc:4841] TS 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665): tablet 02c3ceea1abd414898cf7aa58a030f84 (table test-table [id=053e176d3ad9400c99f82ea9ed4ba53c]) successfully deleted
W20260806 20:38:32.367281 31469 catalog_manager.cc:4808] TS 47c8d6d62daa401dbbb4d66e3a6377a9 (127.23.200.2:33665): delete failed for tablet 02c3ceea1abd414898cf7aa58a030f84 because the tablet was not found. No further retry: Not found: Tablet not found: 02c3ceea1abd414898cf7aa58a030f84
I20260806 20:38:32.368463 32310 ts_tablet_manager.cc:1918] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:32.368614 32310 log.cc:1196] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/02c3ceea1abd414898cf7aa58a030f84
I20260806 20:38:32.369107 32310 ts_tablet_manager.cc:1939] T 02c3ceea1abd414898cf7aa58a030f84 P af44181b22d84f7ab91d75ce7b6a6e0b: Deleting consensus metadata
I20260806 20:38:32.370106 31470 catalog_manager.cc:4841] TS af44181b22d84f7ab91d75ce7b6a6e0b (127.23.200.1:42143): tablet 02c3ceea1abd414898cf7aa58a030f84 (table test-table [id=053e176d3ad9400c99f82ea9ed4ba53c]) successfully deleted
I20260806 20:38:32.372913 32278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.356162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37696 (local address 127.23.200.1:42143)
0806 20:38:32.356311 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.356317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.356733 (+ 416us) server_negotiation.cc:407] Connection header received
0806 20:38:32.356870 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.356875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.356968 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.357086 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:32.359042 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.360214 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.362162 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.362845 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.362990 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.366787 (+ 3797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:32.366806 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:32.366826 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:32.366866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:32.370593 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:32.371180 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:32.371186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:32.371188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:32.371307 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:32.371659 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:32.371662 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:32.371664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:32.371979 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:32.372207 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.372419 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.372572 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 20:38:32.373508 31878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:32.494288 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 31673
I20260806 20:38:32.517654 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 31895
I20260806 20:38:32.531476 32279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.518459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37296 (local address 127.23.200.2:33665)
0806 20:38:32.518660 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.518667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.518685 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:38:32.518902 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.518908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.518983 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.519097 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:32.520500 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.521414 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.522792 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.523214 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.523351 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.526056 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:32.526082 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:32.526099 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:32.526132 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:32.528696 (+ 2564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:32.529359 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:32.529380 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:32.529382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:32.529500 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:32.529864 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:32.529869 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:32.529873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:32.530220 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:32.530579 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.530952 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.531166 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 20:38:32.532207 32032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:32.635329 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 31895
I20260806 20:38:32.654605 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 32050
I20260806 20:38:32.667716 32281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:32.655393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60202 (local address 127.23.200.3:37665)
0806 20:38:32.655606 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:32.655611 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:32.655629 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:38:32.655712 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:32.655716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:32.655776 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:32.655857 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:32.657163 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.657849 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.659213 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:32.659627 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:32.659720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:32.662168 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:32.662182 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:32.662198 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:32.662232 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:32.665151 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:32.665808 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:32.665816 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:32.665818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:32.665950 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:32.666255 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:32.666258 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:32.666260 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:32.666715 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:32.666943 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:32.667191 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 20:38:32.667362 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:38:32.668426 32177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:32.763645 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 32050
I20260806 20:38:32.782979 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 31344
I20260806 20:38:32.784063 31519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:32.911586 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 31344
W20260806 20:38:32.931180 31676 connection.cc:535] server connection from 127.0.0.1:47944 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 20:38:32.931588 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 31449
I20260806 20:38:32.932744 31722 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:33.049588 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 31449
2026-08-06T20:38:33Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9471 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:33 dist-test-slave-tm1g krb5kdc[32323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:33 dist-test-slave-tm1g krb5kdc[32323](info): set up 2 sockets
Aug 06 20:38:33 dist-test-slave-tm1g krb5kdc[32323](info): commencing operation
krb5kdc: starting...
W20260806 20:38:34.162757 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:34 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:34Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab.
Entry for principal oryx/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab.
I20260806 20:38:34.374039 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36913
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab
--principal=oryx/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:36913
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:34.691574 32337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:34.691833 32337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:34.691999 32337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:34.699152 32337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:34.699261 32337 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 20:38:34.699333 32337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:34.699434 32337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:34.699512 32337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:34.699612 32337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:34.709017 32337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:36913
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.62.keytab
--principal=oryx/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:34.710204 32337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:34.711549 32337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:34.723541 32342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:34.723569 32343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:34.723680 32345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:34.724400 32337 server_base.cc:1034] running on GCE node
I20260806 20:38:34.725142 32337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:34.726429 32337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:34.727634 32337 hybrid_clock.cc:648] HybridClock initialized: now 1786048714727566 us; error 45 us; skew 500 ppm
I20260806 20:38:34.727906 32337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:34 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048714, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:34.730959 32337 init.cc:377] Logged in from keytab as oryx/127.23.200.62@KRBTEST.COM (short username oryx)
I20260806 20:38:34.732228 32337 webserver.cc:460] Webserver started at http://127.23.200.62:36523/ using document root <none> and password file <none>
I20260806 20:38:34.732880 32337 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:34.732963 32337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:34.733227 32337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:34.736105 32337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "fb07efa9dfef40ca82e18ea13ec91774"
format_stamp: "Formatted at 2026-08-06 20:38:34 on dist-test-slave-tm1g"
server_key: "6e39040f8f399456a613b3fcb3e29dee"
server_key_iv: "7d0436f7990d5ef4e6e52435e5abad0e"
server_key_version: "encryptionkey@0"
I20260806 20:38:34.736819 32337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "fb07efa9dfef40ca82e18ea13ec91774"
format_stamp: "Formatted at 2026-08-06 20:38:34 on dist-test-slave-tm1g"
server_key: "6e39040f8f399456a613b3fcb3e29dee"
server_key_iv: "7d0436f7990d5ef4e6e52435e5abad0e"
server_key_version: "encryptionkey@0"
I20260806 20:38:34.741830 32337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:38:34.745113 32352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:34.746402 32337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:34.746546 32337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "fb07efa9dfef40ca82e18ea13ec91774"
format_stamp: "Formatted at 2026-08-06 20:38:34 on dist-test-slave-tm1g"
server_key: "6e39040f8f399456a613b3fcb3e29dee"
server_key_iv: "7d0436f7990d5ef4e6e52435e5abad0e"
server_key_version: "encryptionkey@0"
I20260806 20:38:34.746711 32337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:34.770494 32337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:34.774441 32337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:34.774646 32337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:34.796010 32337 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:36913
I20260806 20:38:34.796039 32414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:36913 every 8 connection(s)
I20260806 20:38:34.797299 32337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:34.800882 32415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:34.801878 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 32337
I20260806 20:38:34.802155 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:34.802852 24352 external_mini_cluster.cc:1428] Setting key 44132e25a513be7c8c3999d699c8b7c4
I20260806 20:38:34.814524 32415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Bootstrap starting.
Aug 06 20:38:34 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048714, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:34.818236 32415 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:34.819602 32415 log.cc:824] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:34.823077 32415 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: No bootstrap required, opened a new log
I20260806 20:38:34.826581 32418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:34.804842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.23.200.62:36913)
0806 20:38:34.805722 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:34.805738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:34.805779 (+ 41us) server_negotiation.cc:407] Connection header received
0806 20:38:34.807395 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:34.807431 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:34.807800 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:34.808120 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:34.809882 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:34.811278 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:34.812989 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:34.813608 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:34.813736 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:34.818291 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:34.818319 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:34.818338 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:34.818386 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:34.822082 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:34.822768 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:34.822775 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:34.822783 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:34.823000 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:34.823382 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:34.823387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:34.823390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:34.824101 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:34.824386 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:34.824708 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 20:38:34.825167 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":452,"threads_started":1}
I20260806 20:38:34.832170 32415 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb07efa9dfef40ca82e18ea13ec91774" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36913 } }
I20260806 20:38:34.832476 32415 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:34.832556 32415 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb07efa9dfef40ca82e18ea13ec91774, State: Initialized, Role: FOLLOWER
I20260806 20:38:34.833132 32415 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb07efa9dfef40ca82e18ea13ec91774" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36913 } }
I20260806 20:38:34.833415 32415 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:34.833595 32415 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:34.833770 32415 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:34.835966 32415 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb07efa9dfef40ca82e18ea13ec91774" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36913 } }
I20260806 20:38:34.836434 32415 leader_election.cc:302] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb07efa9dfef40ca82e18ea13ec91774; no voters:
I20260806 20:38:34.837028 32415 leader_election.cc:288] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:34.837150 32420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:34.837965 32420 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [term 1 LEADER]: Becoming Leader. State: Replica: fb07efa9dfef40ca82e18ea13ec91774, State: Running, Role: LEADER
I20260806 20:38:34.838642 32420 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb07efa9dfef40ca82e18ea13ec91774" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36913 } }
I20260806 20:38:34.839552 32415 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:34.841586 32422 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb07efa9dfef40ca82e18ea13ec91774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb07efa9dfef40ca82e18ea13ec91774" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36913 } } }
I20260806 20:38:34.841603 32421 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb07efa9dfef40ca82e18ea13ec91774. Latest consensus state: current_term: 1 leader_uuid: "fb07efa9dfef40ca82e18ea13ec91774" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb07efa9dfef40ca82e18ea13ec91774" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36913 } } }
I20260806 20:38:34.842024 32422 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:34.842092 32421 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:34.843240 32426 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:34.848764 32426 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:34.857376 32426 catalog_manager.cc:1347] Generated new cluster ID: 6c178892720c4e7fa018b770c42628c0
I20260806 20:38:34.857498 32426 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:34.878870 32426 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:34.880411 32426 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:34.887782 32426 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Generated new TSK 0
I20260806 20:38:34.888535 32426 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab.
Entry for principal oryx/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab.
I20260806 20:38:34.935868 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36913
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:35.264870 32442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:35.265132 32442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:35.265228 32442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:35.273270 32442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:35.273371 32442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:35.273464 32442 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:38:35.273542 32442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:35.283202 32442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:35.284487 32442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:35.285832 32442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:35.298880 32452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:35.298964 32448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:35.298930 32450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:35.299963 32442 server_base.cc:1034] running on GCE node
I20260806 20:38:35.300542 32442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:35.301440 32442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:35.302666 32442 hybrid_clock.cc:648] HybridClock initialized: now 1786048715302595 us; error 52 us; skew 500 ppm
I20260806 20:38:35.302956 32442 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:35 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048715, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:35.309860 32442 init.cc:377] Logged in from keytab as oryx/127.23.200.1@KRBTEST.COM (short username oryx)
I20260806 20:38:35.311268 32442 webserver.cc:460] Webserver started at http://127.23.200.1:34791/ using document root <none> and password file <none>
I20260806 20:38:35.311899 32442 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:35.311982 32442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:35.312265 32442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:35.315145 32442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "d0b902f98e254c289f20853e0b916e59"
format_stamp: "Formatted at 2026-08-06 20:38:35 on dist-test-slave-tm1g"
server_key: "34e3e020c9743745a80e2089d03a6fba"
server_key_iv: "681c07ed42c0b3024671fd98c86ebbd6"
server_key_version: "encryptionkey@0"
I20260806 20:38:35.315892 32442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "d0b902f98e254c289f20853e0b916e59"
format_stamp: "Formatted at 2026-08-06 20:38:35 on dist-test-slave-tm1g"
server_key: "34e3e020c9743745a80e2089d03a6fba"
server_key_iv: "681c07ed42c0b3024671fd98c86ebbd6"
server_key_version: "encryptionkey@0"
I20260806 20:38:35.320749 32442 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:38:35.324128 32460 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:35.325366 32442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:38:35.325491 32442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "d0b902f98e254c289f20853e0b916e59"
format_stamp: "Formatted at 2026-08-06 20:38:35 on dist-test-slave-tm1g"
server_key: "34e3e020c9743745a80e2089d03a6fba"
server_key_iv: "681c07ed42c0b3024671fd98c86ebbd6"
server_key_version: "encryptionkey@0"
I20260806 20:38:35.325670 32442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:35.343436 32442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:35.347520 32442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:35.347741 32442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:35.350360 32442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:35.350474 32442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 20:38:35.350642 32442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:35.350756 32442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:38:35 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048715, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:35.373015 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.350742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:43121 (local address 127.23.200.62:36913)
0806 20:38:35.351244 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:35.351252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:35.352333 (+ 1081us) server_negotiation.cc:407] Connection header received
0806 20:38:35.354345 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:35.354352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:35.354442 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:35.354588 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:35.356123 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.357481 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.360003 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.360746 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.360898 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.366086 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:35.366093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:35.366101 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:35.366149 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:35.369573 (+ 3424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.370658 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.370663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.370667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.370816 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.371278 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.371283 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.371287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.371591 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:35.371742 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:35.372402 (+ 660us) server_negotiation.cc:299] Negotiation successful
0806 20:38:35.372654 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":273,"threads_started":1}
I20260806 20:38:35.374033 32471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.351415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.23.200.1:43121)
0806 20:38:35.352177 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:35.352222 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:35.354016 (+ 1794us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:35.354742 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:35.354749 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:35.355275 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:35.355893 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.355925 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.357683 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.357689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.359814 (+ 2125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.359847 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.360997 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.361007 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.361319 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.362016 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:35.362041 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:35.365784 (+ 3743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:35.369728 (+ 3944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.369736 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.369754 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.370475 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.370945 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.370954 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.370958 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.371159 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.371717 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:35.371724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:35.372058 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 20:38:35.372305 (+ 247us) client_negotiation.cc:240] Negotiation successful
0806 20:38:35.372790 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":310,"threads_started":1}
I20260806 20:38:35.394277 32469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.380266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:35049 (local address 127.23.200.62:36913)
0806 20:38:35.380428 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:35.380431 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:35.381095 (+ 664us) server_negotiation.cc:407] Connection header received
0806 20:38:35.381281 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:35.381285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:35.381342 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:35.381435 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:35.382331 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.383214 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.385171 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.385759 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.385897 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.388445 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:35.388452 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:35.388456 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:35.388495 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:35.391433 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.392401 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.392405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.392407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.392536 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.392965 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.392968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.392969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.393205 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:35.393313 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:35.393844 (+ 531us) server_negotiation.cc:299] Negotiation successful
0806 20:38:35.394002 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 20:38:35.394731 32539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.380419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.23.200.1:35049)
0806 20:38:35.380999 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:35.381014 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:35.381158 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:35.381464 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:35.381469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:35.381893 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:35.382167 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.382175 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.383345 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.383350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.384971 (+ 1621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.384991 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.386111 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.386115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.386377 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.387109 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:35.387126 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:35.388267 (+ 1141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:35.391586 (+ 3319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.391594 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.391601 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.392277 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.392644 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.392652 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.392658 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.392853 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.393315 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:35.393323 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:35.393584 (+ 261us) client_negotiation.cc:769] Sending connection context
0806 20:38:35.393775 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 20:38:35.394190 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":322,"threads_started":1}
I20260806 20:38:35.406385 32442 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:35679
I20260806 20:38:35.406468 32584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:35679 every 8 connection(s)
I20260806 20:38:35.407585 32442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:35.409475 32585 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36913
I20260806 20:38:35.409674 32585 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:35.410239 32585 heartbeater.cc:511] Master 127.23.200.62:36913 requested a full tablet report, sending...
I20260806 20:38:35.412106 32370 ts_manager.cc:194] Registered new tserver with Master: d0b902f98e254c289f20853e0b916e59 (127.23.200.1:35679)
I20260806 20:38:35.413486 32370 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:43121
I20260806 20:38:35.416473 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 32442
I20260806 20:38:35.416680 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:35.417294 24352 external_mini_cluster.cc:1428] Setting key 1ec9ca0ae35e1d6f82240aa3fa104590
WARNING: no policy specified for oryx/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab.
Entry for principal oryx/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab.
I20260806 20:38:35.463268 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36913
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:35.785701 32589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:35.785964 32589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:35.786060 32589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:35.793087 32589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:35.793190 32589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:35.793278 32589 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:38:35.793358 32589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:35.802851 32589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:35.804075 32589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:35.805405 32589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:35.818590 32595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:35.818564 32594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:35.818845 32597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:35.819419 32589 server_base.cc:1034] running on GCE node
I20260806 20:38:35.819978 32589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:35.820822 32589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:35.822043 32589 hybrid_clock.cc:648] HybridClock initialized: now 1786048715821980 us; error 31 us; skew 500 ppm
I20260806 20:38:35.822420 32589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:35 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048715, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:35.825380 32589 init.cc:377] Logged in from keytab as oryx/127.23.200.2@KRBTEST.COM (short username oryx)
I20260806 20:38:35.826756 32589 webserver.cc:460] Webserver started at http://127.23.200.2:41617/ using document root <none> and password file <none>
I20260806 20:38:35.827431 32589 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:35.827546 32589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:35.827875 32589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:35.831043 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "4a705afe508b4880b3aba904bae61725"
format_stamp: "Formatted at 2026-08-06 20:38:35 on dist-test-slave-tm1g"
server_key: "f7b6f953f18b2f9bb59cda6a87a875f3"
server_key_iv: "c67ed8b9b37cdc8668434e7de598618a"
server_key_version: "encryptionkey@0"
I20260806 20:38:35.831800 32589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "4a705afe508b4880b3aba904bae61725"
format_stamp: "Formatted at 2026-08-06 20:38:35 on dist-test-slave-tm1g"
server_key: "f7b6f953f18b2f9bb59cda6a87a875f3"
server_key_iv: "c67ed8b9b37cdc8668434e7de598618a"
server_key_version: "encryptionkey@0"
I20260806 20:38:35.836855 32589 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:38:35.840162 32605 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:35.841408 32589 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:38:35.841569 32589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "4a705afe508b4880b3aba904bae61725"
format_stamp: "Formatted at 2026-08-06 20:38:35 on dist-test-slave-tm1g"
server_key: "f7b6f953f18b2f9bb59cda6a87a875f3"
server_key_iv: "c67ed8b9b37cdc8668434e7de598618a"
server_key_version: "encryptionkey@0"
I20260806 20:38:35.841776 32589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:35.918390 32589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:35.922628 32589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:35.922894 32589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:35.925488 32589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:35.925603 32589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:35.925750 32589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:35.925815 32589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:38:35 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048715, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:35.949011 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.926344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:38877 (local address 127.23.200.62:36913)
0806 20:38:35.926833 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:35.926838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:35.927952 (+ 1114us) server_negotiation.cc:407] Connection header received
0806 20:38:35.930126 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:35.930132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:35.930220 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:35.930478 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:35.932562 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.933604 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.936160 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.936892 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.937063 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.941823 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:35.941829 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:35.941849 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:35.941899 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:35.945467 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.946166 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.946171 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.946175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.946383 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.947013 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.947020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.947023 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.947343 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:35.947481 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:35.948396 (+ 915us) server_negotiation.cc:299] Negotiation successful
0806 20:38:35.948633 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":274,"threads_started":1}
I20260806 20:38:35.950011 32615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.926960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.23.200.2:38877)
0806 20:38:35.927794 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:35.927840 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:35.929858 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:35.930733 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:35.930750 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:35.931549 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:35.932383 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.932405 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.933797 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.933801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.935977 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.935998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.937185 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.937194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.937474 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.938182 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:35.938223 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:35.941559 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:35.945600 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.945608 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.945623 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.946041 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.946508 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.946513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.946527 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.946834 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.947473 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:35.947481 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:35.948025 (+ 544us) client_negotiation.cc:769] Sending connection context
0806 20:38:35.948304 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 20:38:35.948800 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":363,"threads_started":1}
I20260806 20:38:35.973019 32691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.956658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.23.200.2:39959)
0806 20:38:35.957453 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:35.957476 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:35.957637 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:35.957955 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:35.957962 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:35.958743 (+ 781us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:35.959455 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.959469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.960494 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.960502 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.962730 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:35.962750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.963462 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.963471 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:35.963708 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.964409 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:35.964433 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:35.965836 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:35.970056 (+ 4220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.970062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.970067 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.970689 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.971165 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:35.971171 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:35.971177 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:35.971374 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:35.972199 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:35.972203 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:35.972361 (+ 158us) client_negotiation.cc:769] Sending connection context
0806 20:38:35.972474 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 20:38:35.972708 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":403,"threads_started":1}
I20260806 20:38:35.973460 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:35.956649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:39959 (local address 127.23.200.62:36913)
0806 20:38:35.956860 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:35.956867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:35.957568 (+ 701us) server_negotiation.cc:407] Connection header received
0806 20:38:35.957760 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:35.957764 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:35.957822 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:35.957891 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:35.959631 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.960353 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.962879 (+ 2526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:35.963267 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:35.963356 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:35.966120 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:35.966127 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:35.966132 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:35.966170 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:35.969874 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.970832 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.970841 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.970847 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.971032 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:35.971533 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:35.971537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:35.971539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:35.971987 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:35.972141 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:35.972680 (+ 539us) server_negotiation.cc:299] Negotiation successful
0806 20:38:35.973008 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:38:35.977675 32589 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33931
I20260806 20:38:35.977722 32733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33931 every 8 connection(s)
I20260806 20:38:35.978919 32589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:35.980855 32734 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36913
I20260806 20:38:35.981048 32734 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:35.981619 32734 heartbeater.cc:511] Master 127.23.200.62:36913 requested a full tablet report, sending...
I20260806 20:38:35.982841 32370 ts_manager.cc:194] Registered new tserver with Master: 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931)
I20260806 20:38:35.983655 32370 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:38877
I20260806 20:38:35.984858 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 32589
I20260806 20:38:35.985059 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:35.985639 24352 external_mini_cluster.cc:1428] Setting key dd9cd379dba105b19fb6f040ad825fd9
WARNING: no policy specified for oryx/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab.
Entry for principal oryx/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab.
I20260806 20:38:36.030901 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36913
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:36.353654 32738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:36.353951 32738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:36.354117 32738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:36.361132 32738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:36.361261 32738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:36.361425 32738 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 20:38:36.361569 32738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:36.370816 32738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.200.62:36913
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:36.372026 32738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:36.373312 32738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:36.385921 32743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:36.385947 32744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:36.386010 32746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:36.387079 32738 server_base.cc:1034] running on GCE node
I20260806 20:38:36.387625 32738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:36.388458 32738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:36.389681 32738 hybrid_clock.cc:648] HybridClock initialized: now 1786048716389623 us; error 38 us; skew 500 ppm
I20260806 20:38:36.389961 32738 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:36 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048716, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:36.392931 32738 init.cc:377] Logged in from keytab as oryx/127.23.200.3@KRBTEST.COM (short username oryx)
I20260806 20:38:36.394276 32738 webserver.cc:460] Webserver started at http://127.23.200.3:37699/ using document root <none> and password file <none>
I20260806 20:38:36.395025 32738 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:36.395149 32738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:36.395512 32738 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:36.398270 32738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "2acb0ecfd6034720b366713b8e8392b4"
format_stamp: "Formatted at 2026-08-06 20:38:36 on dist-test-slave-tm1g"
server_key: "c43818862e4606bafc1036ab6551f3eb"
server_key_iv: "0fd82ccc77f79fd37ff3f31838f9e5e1"
server_key_version: "encryptionkey@0"
I20260806 20:38:36.399139 32738 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "2acb0ecfd6034720b366713b8e8392b4"
format_stamp: "Formatted at 2026-08-06 20:38:36 on dist-test-slave-tm1g"
server_key: "c43818862e4606bafc1036ab6551f3eb"
server_key_iv: "0fd82ccc77f79fd37ff3f31838f9e5e1"
server_key_version: "encryptionkey@0"
I20260806 20:38:36.403980 32738 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:38:36.407661 32753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:36.409138 32738 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:38:36.409296 32738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "2acb0ecfd6034720b366713b8e8392b4"
format_stamp: "Formatted at 2026-08-06 20:38:36 on dist-test-slave-tm1g"
server_key: "c43818862e4606bafc1036ab6551f3eb"
server_key_iv: "0fd82ccc77f79fd37ff3f31838f9e5e1"
server_key_version: "encryptionkey@0"
I20260806 20:38:36.409531 32738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:36.415959 32585 heartbeater.cc:503] Master 127.23.200.62:36913 was elected leader, sending a full tablet report...
I20260806 20:38:36.431442 32738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:36.435051 32738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:36.435303 32738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:36.437780 32738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:36.437860 32738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:36.438059 32738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:36.438169 32738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Aug 06 20:38:36 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048716, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:36.462445 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:36.438246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:33505 (local address 127.23.200.62:36913)
0806 20:38:36.438497 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:36.438506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:36.439973 (+ 1467us) server_negotiation.cc:407] Connection header received
0806 20:38:36.443007 (+ 3034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:36.443013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:36.443112 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:36.443209 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:36.446268 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.447353 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.449923 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.450658 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.450816 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:36.455686 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:36.455694 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:36.455700 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:36.455754 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:36.459144 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:36.460089 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:36.460097 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:36.460102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:36.460241 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:36.460677 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:36.460683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:36.460686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:36.460982 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:36.461135 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:36.461807 (+ 672us) server_negotiation.cc:299] Negotiation successful
0806 20:38:36.462035 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:38:36.463748 32762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:36.438867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.23.200.3:33505)
0806 20:38:36.439795 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:36.439852 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:36.442632 (+ 2780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:36.443836 (+ 1204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:36.443848 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:36.444914 (+ 1066us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:36.446034 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:36.446072 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.447551 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.447555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:36.449762 (+ 2207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:36.449774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.451177 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.451188 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:36.451498 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:36.452124 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:36.452150 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:36.455398 (+ 3248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:36.459285 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:36.459294 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:36.459308 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:36.459959 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:36.460357 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:36.460362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:36.460366 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:36.460567 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:36.461126 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:36.461133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:36.461461 (+ 328us) client_negotiation.cc:769] Sending connection context
0806 20:38:36.461730 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 20:38:36.462208 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":371,"threads_started":1}
I20260806 20:38:36.491017 32738 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:42031
I20260806 20:38:36.491025 408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:42031 every 8 connection(s)
I20260806 20:38:36.492655 32738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:36.494557 382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:36.479574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.23.200.3:43621)
0806 20:38:36.480336 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:36.480374 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:36.480565 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:36.480968 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:36.480971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:36.481414 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:36.481731 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:36.481740 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.483450 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.483453 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:36.484453 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:36.484465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.485590 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.485602 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:36.485833 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:36.486663 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:36.486695 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:36.487672 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:36.491750 (+ 4078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:36.491754 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:36.491757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:36.492139 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:36.492545 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:36.492548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:36.492550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:36.492667 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:36.493349 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:36.493360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:36.493583 (+ 223us) client_negotiation.cc:769] Sending connection context
0806 20:38:36.493748 (+ 165us) client_negotiation.cc:240] Negotiation successful
0806 20:38:36.494073 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":528,"threads_started":1}
I20260806 20:38:36.494836 409 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36913
I20260806 20:38:36.494925 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:36.479017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:43621 (local address 127.23.200.62:36913)
0806 20:38:36.479244 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:36.479250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:36.480460 (+ 1210us) server_negotiation.cc:407] Connection header received
0806 20:38:36.480720 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:36.480729 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:36.480819 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:36.480939 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:36.481935 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.483297 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.484590 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.485362 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.485513 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:36.487881 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:36.487894 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:36.487899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:36.487940 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:36.491635 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:36.492262 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:36.492267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:36.492270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:36.492427 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:36.492797 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:36.492802 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:36.492805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:36.493180 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:36.493345 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:36.493879 (+ 534us) server_negotiation.cc:299] Negotiation successful
0806 20:38:36.494188 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 20:38:36.495113 409 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:36.495991 409 heartbeater.cc:511] Master 127.23.200.62:36913 requested a full tablet report, sending...
I20260806 20:38:36.497555 32370 ts_manager.cc:194] Registered new tserver with Master: 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:36.498677 32370 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:33505
I20260806 20:38:36.500509 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 32738
I20260806 20:38:36.500738 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:36.501154 24352 external_mini_cluster.cc:1428] Setting key ee1232ac046c2c90d63a1c814f7bd9c1
I20260806 20:38:36.503878 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 20:38:36 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048716, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 20:38:36 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:36 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:36.986516 32734 heartbeater.cc:503] Master 127.23.200.62:36913 was elected leader, sending a full tablet report...
I20260806 20:38:37.500994 409 heartbeater.cc:503] Master 127.23.200.62:36913 was elected leader, sending a full tablet report...
W20260806 20:38:37.620172 417 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.072s user 0.004s sys 0.011s
W20260806 20:38:37.621359 32614 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:36.542400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33044 (local address 127.23.200.62:36913)
0806 20:38:36.542624 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:36.542630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:36.542866 (+ 236us) server_negotiation.cc:407] Connection header received
0806 20:38:36.543033 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:36.543037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:36.543109 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:36.543202 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:36.544630 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.545379 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.546719 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:36.547126 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:36.547276 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.621148 (+1073872us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33044: BlockingRecv error: recv got EOF from 127.0.0.1:33044 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 20:38:37 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048716, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:37.641139 32614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.628002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33050 (local address 127.23.200.62:36913)
0806 20:38:37.628187 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:37.628192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:37.628542 (+ 350us) server_negotiation.cc:407] Connection header received
0806 20:38:37.628702 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:37.628706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:37.628773 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:37.628893 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:37.630047 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.630798 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.631942 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.632324 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.632413 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.635961 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:37.635966 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:37.635969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:37.636000 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:37.638649 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:37.639195 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:37.639199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:37.639202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:37.639356 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:37.639729 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:37.639739 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:37.639741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:37.640009 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:37.640286 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:37.640623 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 20:38:37.640833 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:38:37.644809 32370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:37.652264 32370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:37.689036 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.664922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57078 (local address 127.23.200.3:42031)
0806 20:38:37.677745 (+ 12823us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:37.677752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:37.678205 (+ 453us) server_negotiation.cc:407] Connection header received
0806 20:38:37.678444 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:37.678455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:37.679307 (+ 852us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:37.679536 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:37.682777 (+ 3241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.684182 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.686924 (+ 2742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.687898 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.688051 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.688258 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:37.688564 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 20:38:37.688729 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12671,"thread_start_us":12524,"threads_started":1}
I20260806 20:38:37.689085 428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.664242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:42031 (local address 127.0.0.1:57078)
0806 20:38:37.678086 (+ 13844us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:37.678106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:37.678240 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:37.682087 (+ 3847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:37.682093 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:37.682117 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:37.682566 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.682581 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.684323 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.684330 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.686768 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.686780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.688100 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.688106 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.688303 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.688349 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 20:38:37.688470 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 20:38:37.688687 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13750,"mutex_wait_us":1,"thread_start_us":11754,"threads_started":1}
I20260806 20:38:37.690531 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.664077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.23.200.1:35679)
0806 20:38:37.682802 (+ 18725us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:37.682810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:37.682836 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:38:37.682942 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:37.682948 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:37.683173 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:37.683405 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:37.683947 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.685551 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.687936 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.689527 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.689629 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.689754 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:37.690045 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 20:38:37.690192 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18570,"thread_start_us":18409,"threads_started":1}
I20260806 20:38:37.690593 430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.663838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:35679 (local address 127.0.0.1:56942)
0806 20:38:37.676809 (+ 12971us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:37.676856 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:37.677136 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:37.683387 (+ 6251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:37.683393 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:37.683419 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:37.683767 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.683779 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.685677 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.685683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.687764 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.687779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.689715 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.689722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.689874 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.689905 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 20:38:37.690009 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 20:38:37.690158 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11856,"mutex_wait_us":8,"thread_start_us":14051,"threads_started":1}
I20260806 20:38:37.692857 432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.678369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33931 (local address 127.0.0.1:43382)
0806 20:38:37.683186 (+ 4817us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:37.683200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:37.683340 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:37.684102 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:37.684105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:37.684123 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:37.684537 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.684573 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.686451 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.686463 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.688568 (+ 2105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.688582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.692136 (+ 3554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.692142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.692312 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.692342 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 20:38:37.692439 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 20:38:37.692566 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4703,"thread_start_us":4576,"threads_started":1}
I20260806 20:38:37.692976 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.678600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.23.200.2:33931)
0806 20:38:37.682211 (+ 3611us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:37.682222 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:37.683296 (+ 1074us) server_negotiation.cc:407] Connection header received
0806 20:38:37.683544 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:37.683553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:37.683776 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:37.683942 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:37.684759 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.686289 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.688742 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.691686 (+ 2944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.691820 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:37.692000 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:37.692507 (+ 507us) server_negotiation.cc:299] Negotiation successful
0806 20:38:37.692677 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3426,"thread_start_us":3237,"threads_started":1}
I20260806 20:38:37.693238 336 tablet_service.cc:1467] Processing CreateTablet for tablet 81080163957a417dbe8fe7482767a07e (DEFAULT_TABLE table=test-table [id=c76c94d9cc82480099cb614ddbbeec07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:37.693928 32514 tablet_service.cc:1467] Processing CreateTablet for tablet 81080163957a417dbe8fe7482767a07e (DEFAULT_TABLE table=test-table [id=c76c94d9cc82480099cb614ddbbeec07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:37.694833 336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81080163957a417dbe8fe7482767a07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:37.695158 32514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81080163957a417dbe8fe7482767a07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:37.696448 32660 tablet_service.cc:1467] Processing CreateTablet for tablet 81080163957a417dbe8fe7482767a07e (DEFAULT_TABLE table=test-table [id=c76c94d9cc82480099cb614ddbbeec07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:37.697877 32660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81080163957a417dbe8fe7482767a07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:37.707145 434 tablet_bootstrap.cc:492] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Bootstrap starting.
I20260806 20:38:37.709774 437 tablet_bootstrap.cc:492] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Bootstrap starting.
I20260806 20:38:37.710049 434 tablet_bootstrap.cc:654] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:37.711436 434 log.cc:824] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:37.711793 438 tablet_bootstrap.cc:492] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Bootstrap starting.
I20260806 20:38:37.713430 437 tablet_bootstrap.cc:654] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:37.714615 438 tablet_bootstrap.cc:654] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:37.714960 437 log.cc:824] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:37.715865 438 log.cc:824] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:37.717674 434 tablet_bootstrap.cc:492] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: No bootstrap required, opened a new log
I20260806 20:38:37.717808 437 tablet_bootstrap.cc:492] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: No bootstrap required, opened a new log
I20260806 20:38:37.717942 434 ts_tablet_manager.cc:1397] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 20:38:37.718036 437 ts_tablet_manager.cc:1397] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 20:38:37.720213 438 tablet_bootstrap.cc:492] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: No bootstrap required, opened a new log
I20260806 20:38:37.720479 438 ts_tablet_manager.cc:1397] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260806 20:38:37.727711 438 raft_consensus.cc:348] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.728098 438 raft_consensus.cc:374] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:37.728235 438 raft_consensus.cc:729] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a705afe508b4880b3aba904bae61725, State: Initialized, Role: FOLLOWER
I20260806 20:38:37.728713 437 raft_consensus.cc:348] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.728664 434 raft_consensus.cc:348] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.728884 438 consensus_queue.cc:260] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.729084 434 raft_consensus.cc:374] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:37.729084 437 raft_consensus.cc:374] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:37.729228 434 raft_consensus.cc:729] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Initialized, Role: FOLLOWER
I20260806 20:38:37.729247 437 raft_consensus.cc:729] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acb0ecfd6034720b366713b8e8392b4, State: Initialized, Role: FOLLOWER
I20260806 20:38:37.729908 437 consensus_queue.cc:260] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.730435 434 consensus_queue.cc:260] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.732439 434 ts_tablet_manager.cc:1428] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 20:38:37.733821 437 ts_tablet_manager.cc:1428] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Time spent starting tablet: real 0.016s user 0.015s sys 0.000s
I20260806 20:38:37.736012 438 ts_tablet_manager.cc:1428] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
W20260806 20:38:37.748430 410 tablet.cc:2369] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:38:37.911170 32586 tablet.cc:2369] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:38:37.981595 32735 tablet.cc:2369] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:38:37.990587 444 raft_consensus.cc:482] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:37.990880 444 raft_consensus.cc:504] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:37.992478 444 leader_election.cc:288] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031), 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931)
I20260806 20:38:38.001878 433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.993240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:34097 (local address 127.23.200.2:33931)
0806 20:38:37.993456 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:37.993463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:37.993915 (+ 452us) server_negotiation.cc:407] Connection header received
0806 20:38:37.994141 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:37.994147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:37.994223 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:37.994357 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:37.995179 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.996294 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.999455 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.000667 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.000823 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.000895 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.001505 (+ 610us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.001634 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:38:38.001881 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.992977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:36035 (local address 127.23.200.3:42031)
0806 20:38:37.993207 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:37.993213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:37.993992 (+ 779us) server_negotiation.cc:407] Connection header received
0806 20:38:37.994152 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:37.994158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:37.994228 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:37.994349 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:38:37.995179 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.996295 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.999461 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.000679 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.000823 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.000895 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.001507 (+ 612us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.001633 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 20:38:38.002193 446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.993022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:42031 (local address 127.23.200.1:36035)
0806 20:38:37.993894 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:37.993914 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:37.994006 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:37.994372 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:37.994377 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:37.994430 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:37.995033 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.995042 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.996425 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.996431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.999287 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.999299 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.000970 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.000975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.001262 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.001373 (+ 111us) client_negotiation.cc:769] Sending connection context
0806 20:38:38.001469 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 20:38:38.001687 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":755,"mutex_wait_us":308,"spinlock_wait_cycles":28416,"thread_start_us":612,"threads_started":1}
I20260806 20:38:38.002214 445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:37.992912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33931 (local address 127.23.200.1:34097)
0806 20:38:37.993801 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:37.993821 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:37.993987 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:37.994372 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:37.994376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:37.994431 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:38:37.995029 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.995042 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:37.996431 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:37.996435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:37.999284 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:37.999299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.000947 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.000958 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.001266 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.001371 (+ 105us) client_negotiation.cc:769] Sending connection context
0806 20:38:38.001469 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 20:38:38.001687 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"mutex_wait_us":414,"spinlock_wait_cycles":5504,"thread_start_us":454,"threads_started":1}
I20260806 20:38:38.003331 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81080163957a417dbe8fe7482767a07e" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4" is_pre_election: true
I20260806 20:38:38.003372 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81080163957a417dbe8fe7482767a07e" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725" is_pre_election: true
I20260806 20:38:38.003791 359 raft_consensus.cc:2454] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.003804 32681 raft_consensus.cc:2454] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.004412 32461 leader_election.cc:302] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2acb0ecfd6034720b366713b8e8392b4, 4a705afe508b4880b3aba904bae61725, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.004788 444 raft_consensus.cc:2781] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:38.004935 444 raft_consensus.cc:482] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:38.005050 444 raft_consensus.cc:3037] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.007838 444 raft_consensus.cc:504] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:38.008451 444 leader_election.cc:288] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Requested vote from peers 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031), 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931)
I20260806 20:38:38.008864 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81080163957a417dbe8fe7482767a07e" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4"
I20260806 20:38:38.008919 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81080163957a417dbe8fe7482767a07e" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725"
I20260806 20:38:38.009059 359 raft_consensus.cc:3037] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.009186 32681 raft_consensus.cc:3037] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.011266 359 raft_consensus.cc:2454] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.011499 32681 raft_consensus.cc:2454] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.011673 32464 leader_election.cc:302] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acb0ecfd6034720b366713b8e8392b4, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.011974 444 raft_consensus.cc:2781] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:38.012374 444 raft_consensus.cc:686] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Becoming Leader. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Running, Role: LEADER
I20260806 20:38:38.012823 444 consensus_queue.cc:237] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } }
I20260806 20:38:38.017534 32369 catalog_manager.cc:5495] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b902f98e254c289f20853e0b916e59 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:38.042629 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.037498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56956 (local address 127.23.200.1:35679)
0806 20:38:38.037701 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.037707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.037726 (+ 19us) server_negotiation.cc:407] Connection header received
0806 20:38:38.037968 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.037972 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.038043 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.038186 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:38.038569 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.039260 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.040612 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.041010 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.041140 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.041554 (+ 414us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:38.042043 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:38.042189 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.042270 (+ 81us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.042371 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:38:38.050256 359 raft_consensus.cc:1261] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.050367 32681 raft_consensus.cc:1261] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.051307 447 consensus_queue.cc:1035] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.051620 444 consensus_queue.cc:1035] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.061233 452 mvcc.cc:205] Tried to move back new op lower bound from 7315655549125066752 to 7315655548984377344. Current Snapshot: MvccSnapshot[applied={T|T < 7315655549125066752}]
I20260806 20:38:38.075412 32369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:38.075716 32369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:38.079257 32369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Sending DeleteTablet for 3 replicas of tablet 81080163957a417dbe8fe7482767a07e
I20260806 20:38:38.080233 32514 tablet_service.cc:1514] Processing DeleteTablet for tablet 81080163957a417dbe8fe7482767a07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260806 20:38:38.080946 336 tablet_service.cc:1514] Processing DeleteTablet for tablet 81080163957a417dbe8fe7482767a07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:57078
I20260806 20:38:38.081619 32369 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 20:38:38.081928 32660 tablet_service.cc:1514] Processing DeleteTablet for tablet 81080163957a417dbe8fe7482767a07e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:43382
W20260806 20:38:38.082649 32369 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:38.085274 461 tablet_replica.cc:330] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: stopping tablet replica
I20260806 20:38:38.085886 461 raft_consensus.cc:2229] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:38.091804 32513 tablet_service.cc:1467] Processing CreateTablet for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f (DEFAULT_TABLE table=test-table [id=1d517ac572884cd58bcde5eac8db0b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.092432 32513 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.093003 461 raft_consensus.cc:2258] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.094656 336 tablet_service.cc:1467] Processing CreateTablet for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f (DEFAULT_TABLE table=test-table [id=1d517ac572884cd58bcde5eac8db0b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.094810 32659 tablet_service.cc:1467] Processing CreateTablet for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f (DEFAULT_TABLE table=test-table [id=1d517ac572884cd58bcde5eac8db0b55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.095103 336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.095345 32659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.096163 461 ts_tablet_manager.cc:1905] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.100991 462 tablet_replica.cc:330] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: stopping tablet replica
I20260806 20:38:38.101456 462 raft_consensus.cc:2229] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:38.101922 462 raft_consensus.cc:2258] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.102443 438 tablet_bootstrap.cc:492] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Bootstrap starting.
I20260806 20:38:38.104038 438 tablet_bootstrap.cc:654] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.104398 434 tablet_bootstrap.cc:492] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Bootstrap starting.
I20260806 20:38:38.105986 434 tablet_bootstrap.cc:654] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.106429 437 tablet_bootstrap.cc:492] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Bootstrap starting.
I20260806 20:38:38.108007 437 tablet_bootstrap.cc:654] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.110445 461 ts_tablet_manager.cc:1918] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:38.110641 461 log.cc:1196] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/81080163957a417dbe8fe7482767a07e
I20260806 20:38:38.111330 461 ts_tablet_manager.cc:1939] T 81080163957a417dbe8fe7482767a07e P 2acb0ecfd6034720b366713b8e8392b4: Deleting consensus metadata
I20260806 20:38:38.115545 462 ts_tablet_manager.cc:1905] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.115952 32357 catalog_manager.cc:4841] TS 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031): tablet 81080163957a417dbe8fe7482767a07e (table test-table [id=c76c94d9cc82480099cb614ddbbeec07]) successfully deleted
I20260806 20:38:38.116945 438 tablet_bootstrap.cc:492] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: No bootstrap required, opened a new log
I20260806 20:38:38.117095 460 tablet_replica.cc:330] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: stopping tablet replica
I20260806 20:38:38.117100 438 ts_tablet_manager.cc:1397] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.001s
I20260806 20:38:38.117520 460 raft_consensus.cc:2229] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:38.117815 434 tablet_bootstrap.cc:492] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: No bootstrap required, opened a new log
I20260806 20:38:38.117985 434 ts_tablet_manager.cc:1397] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260806 20:38:38.117949 438 raft_consensus.cc:348] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.118201 438 raft_consensus.cc:374] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.118230 460 raft_consensus.cc:2258] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.118922 434 raft_consensus.cc:348] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.119149 434 raft_consensus.cc:374] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.119289 434 raft_consensus.cc:729] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.119618 434 consensus_queue.cc:260] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.121047 460 ts_tablet_manager.cc:1905] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.122171 437 tablet_bootstrap.cc:492] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: No bootstrap required, opened a new log
I20260806 20:38:38.122393 437 ts_tablet_manager.cc:1397] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.000s
I20260806 20:38:38.122551 438 raft_consensus.cc:729] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a705afe508b4880b3aba904bae61725, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.122758 434 ts_tablet_manager.cc:1428] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260806 20:38:38.123227 437 raft_consensus.cc:348] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.123436 437 raft_consensus.cc:374] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.123536 437 raft_consensus.cc:729] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acb0ecfd6034720b366713b8e8392b4, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.123852 437 consensus_queue.cc:260] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.127075 437 ts_tablet_manager.cc:1428] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260806 20:38:38.127204 462 ts_tablet_manager.cc:1918] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:38.127355 462 log.cc:1196] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/81080163957a417dbe8fe7482767a07e
I20260806 20:38:38.127897 462 ts_tablet_manager.cc:1939] T 81080163957a417dbe8fe7482767a07e P 4a705afe508b4880b3aba904bae61725: Deleting consensus metadata
I20260806 20:38:38.129624 32353 catalog_manager.cc:4841] TS 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931): tablet 81080163957a417dbe8fe7482767a07e (table test-table [id=c76c94d9cc82480099cb614ddbbeec07]) successfully deleted
I20260806 20:38:38.129601 438 consensus_queue.cc:260] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.130323 438 ts_tablet_manager.cc:1428] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Time spent starting tablet: real 0.013s user 0.002s sys 0.001s
I20260806 20:38:38.131410 460 ts_tablet_manager.cc:1918] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:38.131587 460 log.cc:1196] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/81080163957a417dbe8fe7482767a07e
I20260806 20:38:38.132145 460 ts_tablet_manager.cc:1939] T 81080163957a417dbe8fe7482767a07e P d0b902f98e254c289f20853e0b916e59: Deleting consensus metadata
I20260806 20:38:38.133719 32353 catalog_manager.cc:4841] TS d0b902f98e254c289f20853e0b916e59 (127.23.200.1:35679): tablet 81080163957a417dbe8fe7482767a07e (table test-table [id=c76c94d9cc82480099cb614ddbbeec07]) successfully deleted
I20260806 20:38:38.271405 457 raft_consensus.cc:482] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:38.271651 457 raft_consensus.cc:504] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.272622 457 leader_election.cc:288] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.273185 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c7f3b7d42ee4b7d91ff0037e6b0384f" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4" is_pre_election: true
I20260806 20:38:38.273231 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c7f3b7d42ee4b7d91ff0037e6b0384f" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725" is_pre_election: true
I20260806 20:38:38.273442 359 raft_consensus.cc:2454] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.273488 32681 raft_consensus.cc:2454] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.273901 32464 leader_election.cc:302] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acb0ecfd6034720b366713b8e8392b4, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.274180 457 raft_consensus.cc:2781] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:38.274331 457 raft_consensus.cc:482] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:38.274451 457 raft_consensus.cc:3037] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.276849 457 raft_consensus.cc:504] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.277446 457 leader_election.cc:288] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Requested vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.277884 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c7f3b7d42ee4b7d91ff0037e6b0384f" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725"
I20260806 20:38:38.277979 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c7f3b7d42ee4b7d91ff0037e6b0384f" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4"
I20260806 20:38:38.278098 32681 raft_consensus.cc:3037] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.278172 359 raft_consensus.cc:3037] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.281071 32681 raft_consensus.cc:2454] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.281071 359 raft_consensus.cc:2454] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.281507 32461 leader_election.cc:302] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2acb0ecfd6034720b366713b8e8392b4, 4a705afe508b4880b3aba904bae61725, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.281999 457 raft_consensus.cc:2781] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:38.282141 457 raft_consensus.cc:686] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Becoming Leader. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Running, Role: LEADER
I20260806 20:38:38.282454 457 consensus_queue.cc:237] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.285887 32370 catalog_manager.cc:5495] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b902f98e254c289f20853e0b916e59 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } health_report { overall_health: UNKNOWN } } }
Aug 06 20:38:38 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048714, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.62@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:38.374964 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.360844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33056 (local address 127.23.200.62:36913)
0806 20:38:38.361774 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.361781 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.361806 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:38:38.361844 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.361847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.361902 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.362015 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:38.363094 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.363882 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.365125 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.365567 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.365711 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.369451 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:38.369456 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:38.369459 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:38.369487 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:38.372126 (+ 2639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:38.373060 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:38.373064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:38.373067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:38.373192 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:38.373619 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:38.373622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:38.373624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:38.373880 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:38.373989 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.374542 (+ 553us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.374707 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":638,"threads_started":1}
I20260806 20:38:38.375468 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.360786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36913 (local address 127.0.0.1:33056)
0806 20:38:38.361465 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:38.361490 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:38.361669 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:38.362032 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:38.362036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:38.362766 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:38.362952 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:38.362961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.364023 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.364031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.364974 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:38.364985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.365747 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.365753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.365960 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.366700 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:38.366717 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:38.369259 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:38.372275 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:38.372289 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:38.372297 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:38.372944 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:38.373310 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:38.373315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:38.373319 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:38.373509 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:38.374018 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:38.374027 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:38.374243 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 20:38:38.374476 (+ 233us) client_negotiation.cc:240] Negotiation successful
0806 20:38:38.374907 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":99,"thread_start_us":332,"threads_started":1}
I20260806 20:38:38.380952 32370 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:33056:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 20:38:38.382046 32370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:38.389201 32514 tablet_service.cc:1467] Processing CreateTablet for tablet 5853334d98324fb9b826dbc2110d8809 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1ebf11a05d449a298b545bc56155723]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:38:38.389550 32659 tablet_service.cc:1467] Processing CreateTablet for tablet 5853334d98324fb9b826dbc2110d8809 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1ebf11a05d449a298b545bc56155723]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:38:38.389853 32514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5853334d98324fb9b826dbc2110d8809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.390090 32659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5853334d98324fb9b826dbc2110d8809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.395241 434 tablet_bootstrap.cc:492] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: Bootstrap starting.
I20260806 20:38:38.395843 336 tablet_service.cc:1467] Processing CreateTablet for tablet 5853334d98324fb9b826dbc2110d8809 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c1ebf11a05d449a298b545bc56155723]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 20:38:38.396432 336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5853334d98324fb9b826dbc2110d8809. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.397122 434 tablet_bootstrap.cc:654] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.397608 438 tablet_bootstrap.cc:492] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: Bootstrap starting.
I20260806 20:38:38.399415 438 tablet_bootstrap.cc:654] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.405973 437 tablet_bootstrap.cc:492] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: Bootstrap starting.
I20260806 20:38:38.407604 437 tablet_bootstrap.cc:654] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.411322 438 tablet_bootstrap.cc:492] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: No bootstrap required, opened a new log
I20260806 20:38:38.411509 438 ts_tablet_manager.cc:1397] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.012s
I20260806 20:38:38.412329 438 raft_consensus.cc:348] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.412602 438 raft_consensus.cc:374] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.412694 438 raft_consensus.cc:729] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a705afe508b4880b3aba904bae61725, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.413017 438 consensus_queue.cc:260] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.413497 442 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.413900 442 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.416951 438 ts_tablet_manager.cc:1428] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260806 20:38:38.418470 437 tablet_bootstrap.cc:492] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: No bootstrap required, opened a new log
I20260806 20:38:38.418642 437 ts_tablet_manager.cc:1397] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260806 20:38:38.419306 437 raft_consensus.cc:348] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.419553 437 raft_consensus.cc:374] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.419631 437 raft_consensus.cc:729] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acb0ecfd6034720b366713b8e8392b4, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.419845 437 consensus_queue.cc:260] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.420260 443 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.420647 443 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.421092 437 ts_tablet_manager.cc:1428] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 20:38:38.423694 434 tablet_bootstrap.cc:492] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: No bootstrap required, opened a new log
I20260806 20:38:38.423856 434 ts_tablet_manager.cc:1397] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.008s
I20260806 20:38:38.424535 434 raft_consensus.cc:348] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.424786 434 raft_consensus.cc:374] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.424854 434 raft_consensus.cc:729] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.425108 434 consensus_queue.cc:260] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.425523 457 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.425800 457 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.426530 434 ts_tablet_manager.cc:1428] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 20:38:38.475633 457 raft_consensus.cc:482] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:38.475912 457 raft_consensus.cc:504] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.476786 457 leader_election.cc:288] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.477383 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853334d98324fb9b826dbc2110d8809" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725" is_pre_election: true
I20260806 20:38:38.477588 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853334d98324fb9b826dbc2110d8809" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4" is_pre_election: true
I20260806 20:38:38.477690 32681 raft_consensus.cc:2454] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.477874 359 raft_consensus.cc:2454] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.478252 32461 leader_election.cc:302] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a705afe508b4880b3aba904bae61725, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.478663 457 raft_consensus.cc:2781] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:38.478780 457 raft_consensus.cc:482] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:38.478910 457 raft_consensus.cc:3037] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.482046 457 raft_consensus.cc:504] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.483070 457 leader_election.cc:288] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Requested vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.483075 32681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853334d98324fb9b826dbc2110d8809" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725"
I20260806 20:38:38.483150 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5853334d98324fb9b826dbc2110d8809" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4"
I20260806 20:38:38.483295 32681 raft_consensus.cc:3037] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.483374 359 raft_consensus.cc:3037] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.486335 359 raft_consensus.cc:2454] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.486341 32681 raft_consensus.cc:2454] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.486766 32461 leader_election.cc:302] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a705afe508b4880b3aba904bae61725, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.487047 457 raft_consensus.cc:2781] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:38.487231 457 raft_consensus.cc:686] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Becoming Leader. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Running, Role: LEADER
I20260806 20:38:38.487488 457 consensus_queue.cc:237] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.489248 444 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: TxnStatusTablet state changed. Reason: New leader d0b902f98e254c289f20853e0b916e59. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.489545 444 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:38:38.490248 480 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 20:38:38.490515 480 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 20:38:38.491026 32369 catalog_manager.cc:5495] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b902f98e254c289f20853e0b916e59 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:38.526011 473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.520504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:35679 (local address 127.0.0.1:42856)
0806 20:38:38.520736 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:38.520755 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:38.520865 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:38.521244 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:38.521247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:38.521256 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:38:38.521421 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:38.521428 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.522858 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.522862 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.523941 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:38.523953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.524876 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.524880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.525026 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.525075 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:38:38.525484 (+ 409us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:38:38.525506 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:38:38.525604 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 20:38:38.525773 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 20:38:38.526037 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.520625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.23.200.1:35679)
0806 20:38:38.520811 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.520818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.520838 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:38:38.521031 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.521037 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.521120 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.521205 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:38.521534 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.522733 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.524070 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.524743 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.524876 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.525230 (+ 354us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:38.525382 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:38.525464 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.525630 (+ 166us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.525708 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:38:38.529552 32681 raft_consensus.cc:1261] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.529633 359 raft_consensus.cc:1261] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.530200 444 consensus_queue.cc:1035] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.530702 457 consensus_queue.cc:1035] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.532284 442 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: TxnStatusTablet state changed. Reason: New leader d0b902f98e254c289f20853e0b916e59. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.532558 442 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.534264 447 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.534525 447 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:38:38.534722 450 mvcc.cc:205] Tried to move back new op lower bound from 7315655551090991104 to 7315655550927093760. Current Snapshot: MvccSnapshot[applied={T|T < 7315655551090991104}]
I20260806 20:38:38.535699 451 mvcc.cc:205] Tried to move back new op lower bound from 7315655551090991104 to 7315655550927093760. Current Snapshot: MvccSnapshot[applied={T|T < 7315655551090991104 or (T in {7315655551090991104})}]
I20260806 20:38:38.537397 442 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.537601 442 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P 4a705afe508b4880b3aba904bae61725: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.544236 444 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.544525 444 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P d0b902f98e254c289f20853e0b916e59: This TxnStatusTablet replica's current role is: LEADER
I20260806 20:38:38.544365 443 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: TxnStatusTablet state changed. Reason: New leader d0b902f98e254c289f20853e0b916e59. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.544616 443 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.547628 443 tablet_replica.cc:439] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } } }
I20260806 20:38:38.548241 443 tablet_replica.cc:442] T 5853334d98324fb9b826dbc2110d8809 P 2acb0ecfd6034720b366713b8e8392b4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 20:38:38.561820 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.555302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:46353 (local address 127.23.200.1:35679)
0806 20:38:38.555476 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.555486 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.555885 (+ 399us) server_negotiation.cc:407] Connection header received
0806 20:38:38.556039 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.556044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.556128 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.556233 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:38.556823 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.558062 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.559756 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.560436 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.560572 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.560970 (+ 398us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:38.561137 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:38.561268 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.561422 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.561514 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":62}
I20260806 20:38:38.561846 486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.555102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:35679 (local address 127.23.200.1:46353)
0806 20:38:38.555786 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:38.555805 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:38.555921 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:38.556274 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:38.556278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:38.556288 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 20:38:38.556659 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:38.556673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.558217 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.558221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.559615 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:38.559627 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.560588 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.560592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:38.560742 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.560791 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 20:38:38.561277 (+ 486us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 20:38:38.561293 (+ 16us) client_negotiation.cc:769] Sending connection context
0806 20:38:38.561377 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 20:38:38.561486 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":2,"thread_start_us":345,"threads_started":1}
I20260806 20:38:38.570945 359 raft_consensus.cc:1261] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.571033 32681 raft_consensus.cc:1261] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.571535 444 consensus_queue.cc:1035] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.571861 457 consensus_queue.cc:1035] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.625211 32368 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:38.625438 32368 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:38.628688 32368 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Sending DeleteTablet for 3 replicas of tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f
I20260806 20:38:38.629515 32659 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:43382
I20260806 20:38:38.629698 336 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:57078
I20260806 20:38:38.630028 32514 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260806 20:38:38.630167 490 tablet_replica.cc:330] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: stopping tablet replica
I20260806 20:38:38.630318 460 tablet_replica.cc:330] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: stopping tablet replica
I20260806 20:38:38.630424 490 raft_consensus.cc:2229] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:38.630523 460 raft_consensus.cc:2229] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:38.630640 490 raft_consensus.cc:2258] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.630847 460 raft_consensus.cc:2258] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.631721 490 ts_tablet_manager.cc:1905] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.632037 460 ts_tablet_manager.cc:1905] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.639515 490 ts_tablet_manager.cc:1918] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:38:38.639711 490 log.cc:1196] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/7c7f3b7d42ee4b7d91ff0037e6b0384f
I20260806 20:38:38.640313 490 ts_tablet_manager.cc:1939] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 4a705afe508b4880b3aba904bae61725: Deleting consensus metadata
I20260806 20:38:38.640786 491 tablet_replica.cc:330] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: stopping tablet replica
I20260806 20:38:38.641072 491 raft_consensus.cc:2229] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:38.641316 491 raft_consensus.cc:2258] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.641649 32353 catalog_manager.cc:4841] TS 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931): tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f (table test-table [id=1d517ac572884cd58bcde5eac8db0b55]) successfully deleted
I20260806 20:38:38.642709 491 ts_tablet_manager.cc:1905] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.648082 460 ts_tablet_manager.cc:1918] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:38:38.648386 460 log.cc:1196] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/7c7f3b7d42ee4b7d91ff0037e6b0384f
I20260806 20:38:38.649855 491 ts_tablet_manager.cc:1918] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:38:38.649973 460 ts_tablet_manager.cc:1939] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P d0b902f98e254c289f20853e0b916e59: Deleting consensus metadata
I20260806 20:38:38.650025 491 log.cc:1196] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/7c7f3b7d42ee4b7d91ff0037e6b0384f
I20260806 20:38:38.650465 491 ts_tablet_manager.cc:1939] T 7c7f3b7d42ee4b7d91ff0037e6b0384f P 2acb0ecfd6034720b366713b8e8392b4: Deleting consensus metadata
I20260806 20:38:38.651172 32353 catalog_manager.cc:4841] TS d0b902f98e254c289f20853e0b916e59 (127.23.200.1:35679): tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f (table test-table [id=1d517ac572884cd58bcde5eac8db0b55]) successfully deleted
I20260806 20:38:38.651535 32357 catalog_manager.cc:4841] TS 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031): tablet 7c7f3b7d42ee4b7d91ff0037e6b0384f (table test-table [id=1d517ac572884cd58bcde5eac8db0b55]) successfully deleted
Aug 06 20:38:38 dist-test-slave-tm1g krb5kdc[32323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 20:38:38 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:38 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 20:38:38.696058 474 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:38.685039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.23.200.62:36913)
0806 20:38:38.685313 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.685323 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.685892 (+ 569us) server_negotiation.cc:407] Connection header received
0806 20:38:38.686117 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.686121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.686204 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.686327 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:38.687670 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.688433 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.689684 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.690124 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.690263 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.695795 (+ 5532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38816: BlockingRecv error: recv got EOF from 127.0.0.1:38816 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 20:38:38 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM
I20260806 20:38:38.714095 474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.699104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.23.200.62:36913)
0806 20:38:38.699323 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.699332 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.699671 (+ 339us) server_negotiation.cc:407] Connection header received
0806 20:38:38.699900 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.699907 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.699996 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.700116 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:38.701575 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.702281 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.704117 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.704529 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.704620 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.708893 (+ 4273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:38.708898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:38.708901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:38.708933 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:38.711501 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:38.712151 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:38.712161 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:38.712164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:38.712286 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:38.712639 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:38.712644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:38.712647 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:38.712924 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:38.713194 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.713502 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.713732 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 20:38:38.717895 32368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:38.718644 32368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:38.725914 32514 tablet_service.cc:1467] Processing CreateTablet for tablet 380a816da1864278964ae8a3a909e5e7 (DEFAULT_TABLE table=test-table [id=4dbee57346b24d069d92064d1d6a1f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.726207 336 tablet_service.cc:1467] Processing CreateTablet for tablet 380a816da1864278964ae8a3a909e5e7 (DEFAULT_TABLE table=test-table [id=4dbee57346b24d069d92064d1d6a1f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.726441 32514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380a816da1864278964ae8a3a909e5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.726228 32659 tablet_service.cc:1467] Processing CreateTablet for tablet 380a816da1864278964ae8a3a909e5e7 (DEFAULT_TABLE table=test-table [id=4dbee57346b24d069d92064d1d6a1f16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.726683 336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380a816da1864278964ae8a3a909e5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.726830 32659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 380a816da1864278964ae8a3a909e5e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.733512 434 tablet_bootstrap.cc:492] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Bootstrap starting.
I20260806 20:38:38.733615 437 tablet_bootstrap.cc:492] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Bootstrap starting.
I20260806 20:38:38.734848 437 tablet_bootstrap.cc:654] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.735230 434 tablet_bootstrap.cc:654] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.736765 437 tablet_bootstrap.cc:492] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: No bootstrap required, opened a new log
I20260806 20:38:38.736917 437 ts_tablet_manager.cc:1397] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260806 20:38:38.737582 437 raft_consensus.cc:348] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.737813 437 raft_consensus.cc:374] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.737905 437 raft_consensus.cc:729] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acb0ecfd6034720b366713b8e8392b4, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.738124 437 consensus_queue.cc:260] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.739272 437 ts_tablet_manager.cc:1428] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260806 20:38:38.739873 438 tablet_bootstrap.cc:492] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Bootstrap starting.
I20260806 20:38:38.741406 438 tablet_bootstrap.cc:654] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.744382 438 tablet_bootstrap.cc:492] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: No bootstrap required, opened a new log
I20260806 20:38:38.744555 438 ts_tablet_manager.cc:1397] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260806 20:38:38.745610 434 tablet_bootstrap.cc:492] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: No bootstrap required, opened a new log
I20260806 20:38:38.745438 438 raft_consensus.cc:348] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.745687 438 raft_consensus.cc:374] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.745797 438 raft_consensus.cc:729] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a705afe508b4880b3aba904bae61725, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.745836 434 ts_tablet_manager.cc:1397] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260806 20:38:38.746292 438 consensus_queue.cc:260] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.746932 438 ts_tablet_manager.cc:1428] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 20:38:38.747363 434 raft_consensus.cc:348] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.747624 434 raft_consensus.cc:374] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.747715 434 raft_consensus.cc:729] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.747975 434 consensus_queue.cc:260] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.748517 434 ts_tablet_manager.cc:1428] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260806 20:38:38.760206 444 raft_consensus.cc:482] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:38.760380 444 raft_consensus.cc:504] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.761013 444 leader_election.cc:288] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.761447 32679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a816da1864278964ae8a3a909e5e7" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725" is_pre_election: true
I20260806 20:38:38.761554 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a816da1864278964ae8a3a909e5e7" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4" is_pre_election: true
I20260806 20:38:38.761719 32679 raft_consensus.cc:2454] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.761778 359 raft_consensus.cc:2454] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.762192 32461 leader_election.cc:302] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a705afe508b4880b3aba904bae61725, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.762519 444 raft_consensus.cc:2781] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:38.762652 444 raft_consensus.cc:482] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:38.762768 444 raft_consensus.cc:3037] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.764955 444 raft_consensus.cc:504] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.765548 444 leader_election.cc:288] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Requested vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.766016 32679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a816da1864278964ae8a3a909e5e7" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725"
I20260806 20:38:38.766204 32679 raft_consensus.cc:3037] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.766127 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "380a816da1864278964ae8a3a909e5e7" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4"
I20260806 20:38:38.766367 359 raft_consensus.cc:3037] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.768514 32679 raft_consensus.cc:2454] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.768934 32461 leader_election.cc:302] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4a705afe508b4880b3aba904bae61725, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.769227 444 raft_consensus.cc:2781] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:38.769423 444 raft_consensus.cc:686] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Becoming Leader. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Running, Role: LEADER
I20260806 20:38:38.769706 444 consensus_queue.cc:237] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.769389 359 raft_consensus.cc:2454] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.772388 32370 catalog_manager.cc:5495] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b902f98e254c289f20853e0b916e59 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:38.788116 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:38.782528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.23.200.1:35679)
0806 20:38:38.782704 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:38.782708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:38.782815 (+ 107us) server_negotiation.cc:407] Connection header received
0806 20:38:38.783013 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:38.783017 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:38.783080 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:38.783153 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:38:38.783564 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.784338 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.786260 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:38.786788 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:38.786892 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:38.787411 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:38:38.787522 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:38:38.787586 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:38.787748 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 20:38:38.787817 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:38:38.790622 359 raft_consensus.cc:1261] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.790666 32679 raft_consensus.cc:1261] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.791234 457 consensus_queue.cc:1035] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.791488 444 consensus_queue.cc:1035] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.809968 32370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:38.810227 32370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:38.813544 32370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Sending DeleteTablet for 3 replicas of tablet 380a816da1864278964ae8a3a909e5e7
I20260806 20:38:38.814479 32514 tablet_service.cc:1514] Processing DeleteTablet for tablet 380a816da1864278964ae8a3a909e5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260806 20:38:38.814652 32659 tablet_service.cc:1514] Processing DeleteTablet for tablet 380a816da1864278964ae8a3a909e5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:43382
I20260806 20:38:38.814859 460 tablet_replica.cc:330] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: stopping tablet replica
I20260806 20:38:38.814944 490 tablet_replica.cc:330] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: stopping tablet replica
I20260806 20:38:38.815047 460 raft_consensus.cc:2229] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:38.815169 490 raft_consensus.cc:2229] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:38.815359 460 raft_consensus.cc:2258] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.815378 490 raft_consensus.cc:2258] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.816259 336 tablet_service.cc:1514] Processing DeleteTablet for tablet 380a816da1864278964ae8a3a909e5e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:38 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:57078
I20260806 20:38:38.816434 460 ts_tablet_manager.cc:1905] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.816533 491 tablet_replica.cc:330] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: stopping tablet replica
I20260806 20:38:38.816727 491 raft_consensus.cc:2229] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:38.816943 491 raft_consensus.cc:2258] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:38.819310 490 ts_tablet_manager.cc:1905] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.823316 460 ts_tablet_manager.cc:1918] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:38.823455 460 log.cc:1196] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/380a816da1864278964ae8a3a909e5e7
I20260806 20:38:38.823858 460 ts_tablet_manager.cc:1939] T 380a816da1864278964ae8a3a909e5e7 P d0b902f98e254c289f20853e0b916e59: Deleting consensus metadata
I20260806 20:38:38.824925 32353 catalog_manager.cc:4841] TS d0b902f98e254c289f20853e0b916e59 (127.23.200.1:35679): tablet 380a816da1864278964ae8a3a909e5e7 (table test-table [id=4dbee57346b24d069d92064d1d6a1f16]) successfully deleted
I20260806 20:38:38.825389 32370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:38:38.826032 32370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:38:38.826200 490 ts_tablet_manager.cc:1918] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:38.827557 491 ts_tablet_manager.cc:1905] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:38.831130 490 log.cc:1196] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/380a816da1864278964ae8a3a909e5e7
I20260806 20:38:38.831609 490 ts_tablet_manager.cc:1939] T 380a816da1864278964ae8a3a909e5e7 P 4a705afe508b4880b3aba904bae61725: Deleting consensus metadata
I20260806 20:38:38.832664 32353 catalog_manager.cc:4841] TS 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931): tablet 380a816da1864278964ae8a3a909e5e7 (table test-table [id=4dbee57346b24d069d92064d1d6a1f16]) successfully deleted
I20260806 20:38:38.834784 491 ts_tablet_manager.cc:1918] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:38:38.834960 491 log.cc:1196] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/380a816da1864278964ae8a3a909e5e7
I20260806 20:38:38.835448 491 ts_tablet_manager.cc:1939] T 380a816da1864278964ae8a3a909e5e7 P 2acb0ecfd6034720b366713b8e8392b4: Deleting consensus metadata
I20260806 20:38:38.836242 32514 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfbafe43aec4bc0bca2426ea358c862 (DEFAULT_TABLE table=test-table [id=11932e81ccd547278dcc36e32985d57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.836426 336 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfbafe43aec4bc0bca2426ea358c862 (DEFAULT_TABLE table=test-table [id=11932e81ccd547278dcc36e32985d57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.836782 32514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfbafe43aec4bc0bca2426ea358c862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.836836 336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfbafe43aec4bc0bca2426ea358c862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.837042 32659 tablet_service.cc:1467] Processing CreateTablet for tablet 2bfbafe43aec4bc0bca2426ea358c862 (DEFAULT_TABLE table=test-table [id=11932e81ccd547278dcc36e32985d57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:38:38.837414 32659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bfbafe43aec4bc0bca2426ea358c862. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:38.842590 32357 catalog_manager.cc:4841] TS 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031): tablet 380a816da1864278964ae8a3a909e5e7 (table test-table [id=4dbee57346b24d069d92064d1d6a1f16]) successfully deleted
I20260806 20:38:38.843545 437 tablet_bootstrap.cc:492] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Bootstrap starting.
I20260806 20:38:38.844063 438 tablet_bootstrap.cc:492] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Bootstrap starting.
I20260806 20:38:38.844220 434 tablet_bootstrap.cc:492] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Bootstrap starting.
I20260806 20:38:38.845178 437 tablet_bootstrap.cc:654] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.845695 438 tablet_bootstrap.cc:654] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.846014 434 tablet_bootstrap.cc:654] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:38.854395 438 tablet_bootstrap.cc:492] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: No bootstrap required, opened a new log
I20260806 20:38:38.854559 438 ts_tablet_manager.cc:1397] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 20:38:38.854579 434 tablet_bootstrap.cc:492] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: No bootstrap required, opened a new log
I20260806 20:38:38.854835 434 ts_tablet_manager.cc:1397] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20260806 20:38:38.855434 438 raft_consensus.cc:348] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.855698 438 raft_consensus.cc:374] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.855796 438 raft_consensus.cc:729] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a705afe508b4880b3aba904bae61725, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.855736 434 raft_consensus.cc:348] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.856187 434 raft_consensus.cc:374] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.856069 438 consensus_queue.cc:260] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.856338 434 raft_consensus.cc:729] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.856638 434 consensus_queue.cc:260] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.857303 434 ts_tablet_manager.cc:1428] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 20:38:38.857827 438 ts_tablet_manager.cc:1428] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 20:38:38.858850 437 tablet_bootstrap.cc:492] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: No bootstrap required, opened a new log
I20260806 20:38:38.859094 437 ts_tablet_manager.cc:1397] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.005s
I20260806 20:38:38.859948 437 raft_consensus.cc:348] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.860260 437 raft_consensus.cc:374] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:38.860396 437 raft_consensus.cc:729] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2acb0ecfd6034720b366713b8e8392b4, State: Initialized, Role: FOLLOWER
I20260806 20:38:38.860709 437 consensus_queue.cc:260] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.861320 437 ts_tablet_manager.cc:1428] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 20:38:38.924335 444 raft_consensus.cc:482] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:38:38.924536 444 raft_consensus.cc:504] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.925194 444 leader_election.cc:288] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.925755 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfbafe43aec4bc0bca2426ea358c862" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4" is_pre_election: true
I20260806 20:38:38.925778 32679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfbafe43aec4bc0bca2426ea358c862" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725" is_pre_election: true
I20260806 20:38:38.925994 359 raft_consensus.cc:2454] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.926074 32679 raft_consensus.cc:2454] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 0.
I20260806 20:38:38.926481 32464 leader_election.cc:302] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acb0ecfd6034720b366713b8e8392b4, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.926736 444 raft_consensus.cc:2781] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:38:38.926877 444 raft_consensus.cc:482] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:38:38.926993 444 raft_consensus.cc:3037] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.929323 444 raft_consensus.cc:504] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.929883 444 leader_election.cc:288] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Requested vote from peers 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931), 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031)
I20260806 20:38:38.930368 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfbafe43aec4bc0bca2426ea358c862" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2acb0ecfd6034720b366713b8e8392b4"
I20260806 20:38:38.930393 32679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bfbafe43aec4bc0bca2426ea358c862" candidate_uuid: "d0b902f98e254c289f20853e0b916e59" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a705afe508b4880b3aba904bae61725"
I20260806 20:38:38.930572 359 raft_consensus.cc:3037] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.930584 32679 raft_consensus.cc:3037] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:38.932665 359 raft_consensus.cc:2454] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.933176 32464 leader_election.cc:302] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2acb0ecfd6034720b366713b8e8392b4, d0b902f98e254c289f20853e0b916e59; no voters:
I20260806 20:38:38.933521 444 raft_consensus.cc:2781] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:38.933632 32679 raft_consensus.cc:2454] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0b902f98e254c289f20853e0b916e59 in term 1.
I20260806 20:38:38.934147 444 raft_consensus.cc:686] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Becoming Leader. State: Replica: d0b902f98e254c289f20853e0b916e59, State: Running, Role: LEADER
I20260806 20:38:38.934451 444 consensus_queue.cc:237] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } }
I20260806 20:38:38.937105 32369 catalog_manager.cc:5495] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 reported cstate change: term changed from 0 to 1, leader changed from <none> to d0b902f98e254c289f20853e0b916e59 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "d0b902f98e254c289f20853e0b916e59" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0b902f98e254c289f20853e0b916e59" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 35679 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 } health_report { overall_health: UNKNOWN } } }
I20260806 20:38:38.973197 359 raft_consensus.cc:1261] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.973371 32679 raft_consensus.cc:1261] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Refusing update from remote peer d0b902f98e254c289f20853e0b916e59: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:38:38.973819 444 consensus_queue.cc:1035] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2acb0ecfd6034720b366713b8e8392b4" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 42031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:38.974162 447 consensus_queue.cc:1035] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a705afe508b4880b3aba904bae61725" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:38:39.040568 32370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:39.040853 32370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38828:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:38:39.043601 32370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb07efa9dfef40ca82e18ea13ec91774: Sending DeleteTablet for 3 replicas of tablet 2bfbafe43aec4bc0bca2426ea358c862
I20260806 20:38:39.044317 32513 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfbafe43aec4bc0bca2426ea358c862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:39 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56942
I20260806 20:38:39.044453 336 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfbafe43aec4bc0bca2426ea358c862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:39 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:57078
I20260806 20:38:39.044467 32659 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bfbafe43aec4bc0bca2426ea358c862 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:38:39 UTC) from {username='oryx', principal='oryx/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:43382
I20260806 20:38:39.044622 460 tablet_replica.cc:330] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: stopping tablet replica
I20260806 20:38:39.044767 491 tablet_replica.cc:330] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: stopping tablet replica
I20260806 20:38:39.044850 460 raft_consensus.cc:2229] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:38:39.044984 491 raft_consensus.cc:2229] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:39.045197 491 raft_consensus.cc:2258] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:39.045212 460 raft_consensus.cc:2258] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:39.045552 490 tablet_replica.cc:330] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: stopping tablet replica
I20260806 20:38:39.045747 490 raft_consensus.cc:2229] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:38:39.045958 490 raft_consensus.cc:2258] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:38:39.046166 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 32442
I20260806 20:38:39.046283 460 ts_tablet_manager.cc:1905] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:39.046453 491 ts_tablet_manager.cc:1905] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:39.048669 490 ts_tablet_manager.cc:1905] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:38:39.053574 491 ts_tablet_manager.cc:1918] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:38:39.053692 491 log.cc:1196] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/2bfbafe43aec4bc0bca2426ea358c862
I20260806 20:38:39.054152 491 ts_tablet_manager.cc:1939] T 2bfbafe43aec4bc0bca2426ea358c862 P 2acb0ecfd6034720b366713b8e8392b4: Deleting consensus metadata
I20260806 20:38:39.055148 490 ts_tablet_manager.cc:1918] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:38:39.055217 460 ts_tablet_manager.cc:1918] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 20:38:39.055279 490 log.cc:1196] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/2bfbafe43aec4bc0bca2426ea358c862
I20260806 20:38:39.055400 460 log.cc:1196] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/2bfbafe43aec4bc0bca2426ea358c862
I20260806 20:38:39.055421 32357 catalog_manager.cc:4841] TS 2acb0ecfd6034720b366713b8e8392b4 (127.23.200.3:42031): tablet 2bfbafe43aec4bc0bca2426ea358c862 (table test-table [id=11932e81ccd547278dcc36e32985d57b]) successfully deleted
I20260806 20:38:39.055701 490 ts_tablet_manager.cc:1939] T 2bfbafe43aec4bc0bca2426ea358c862 P 4a705afe508b4880b3aba904bae61725: Deleting consensus metadata
I20260806 20:38:39.055884 460 ts_tablet_manager.cc:1939] T 2bfbafe43aec4bc0bca2426ea358c862 P d0b902f98e254c289f20853e0b916e59: Deleting consensus metadata
I20260806 20:38:39.056593 32353 catalog_manager.cc:4841] TS 4a705afe508b4880b3aba904bae61725 (127.23.200.2:33931): tablet 2bfbafe43aec4bc0bca2426ea358c862 (table test-table [id=11932e81ccd547278dcc36e32985d57b]) successfully deleted
I20260806 20:38:39.056952 32353 catalog_manager.cc:4841] TS d0b902f98e254c289f20853e0b916e59 (127.23.200.1:35679): tablet 2bfbafe43aec4bc0bca2426ea358c862 (table test-table [id=11932e81ccd547278dcc36e32985d57b]) successfully deleted
Aug 06 20:38:39 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.1@KRBTEST.COM
I20260806 20:38:39.070165 429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:39.047418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.23.200.1:35679)
0806 20:38:39.047617 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:39.047623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:39.057071 (+ 9448us) server_negotiation.cc:407] Connection header received
0806 20:38:39.057235 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:39.057238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:39.057304 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:39.057384 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:39.058623 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:39.059278 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:39.060424 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:39.060832 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:39.060918 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:39.064828 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:39.064843 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:39.064859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:39.064888 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:39.067751 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:39.068510 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:39.068517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:39.068520 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:39.068652 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:39.068964 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:39.068967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:39.068969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:39.069311 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:39.069523 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:39.069767 (+ 244us) server_negotiation.cc:299] Negotiation successful
0806 20:38:39.069939 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 20:38:39.070840 32579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:39.218685 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 32442
I20260806 20:38:39.247197 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 32589
Aug 06 20:38:39 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.2@KRBTEST.COM
I20260806 20:38:39.262012 521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:39.247956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36920 (local address 127.23.200.2:33931)
0806 20:38:39.248527 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:39.248531 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:39.248564 (+ 33us) server_negotiation.cc:407] Connection header received
0806 20:38:39.248633 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:39.248636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:39.248695 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:39.248791 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:39.250176 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:39.251002 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:39.252274 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:39.252790 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:39.252887 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:39.256841 (+ 3954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:39.256854 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:39.256871 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:39.256900 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:39.259733 (+ 2833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:39.260302 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:39.260308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:39.260310 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:39.260429 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:39.260709 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:39.260712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:39.260714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:39.261051 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:39.261271 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:39.261470 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 20:38:39.261629 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":275,"threads_started":1}
I20260806 20:38:39.262696 32729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:39.361908 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 32589
I20260806 20:38:39.380897 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 32738
Aug 06 20:38:39 dist-test-slave-tm1g krb5kdc[32323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048718, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.200.3@KRBTEST.COM
I20260806 20:38:39.396840 523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:39.381685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.23.200.3:42031)
0806 20:38:39.382180 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:39.382184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:39.382199 (+ 15us) server_negotiation.cc:407] Connection header received
0806 20:38:39.382253 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:39.382257 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:39.382416 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:39.382535 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:39.384101 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:39.384862 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:39.386127 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:39.386678 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:39.386816 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:39.391215 (+ 4399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:39.391231 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:39.391248 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:39.391281 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:39.394013 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:39.394733 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:39.394740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:39.394742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:39.394877 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:39.395248 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:39.395254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:39.395257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:39.395628 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:39.395963 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:39.396242 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 20:38:39.396506 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":256,"threads_started":1}
I20260806 20:38:39.397509 404 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:39.491326 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 32738
I20260806 20:38:39.510125 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 32337
I20260806 20:38:39.511332 32410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:39.628217 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 32337
2026-08-06T20:38:39Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6577 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:39 dist-test-slave-tm1g krb5kdc[528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:39 dist-test-slave-tm1g krb5kdc[528](info): set up 2 sockets
Aug 06 20:38:39 dist-test-slave-tm1g krb5kdc[528](info): commencing operation
krb5kdc: starting...
W20260806 20:38:40.774580 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:40 dist-test-slave-tm1g krb5kdc[528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048720, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:38:40.982173 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35137
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:35505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:41.299485 541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:41.299743 541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:41.299902 541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:41.306931 541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:41.307024 541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:41.307124 541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:41.307205 541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:41.307271 541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:41.316671 541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:35505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35137
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:41.317893 541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:41.319195 541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:41.330684 549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:41.330801 546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:41.330809 547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:41.332046 541 server_base.cc:1034] running on GCE node
I20260806 20:38:41.332782 541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:41.334105 541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:41.335359 541 hybrid_clock.cc:648] HybridClock initialized: now 1786048721335312 us; error 47 us; skew 500 ppm
I20260806 20:38:41.335717 541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T20:38:41Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1775 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:38:41 dist-test-slave-tm1g krb5kdc[556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:38:41 dist-test-slave-tm1g krb5kdc[556](info): set up 2 sockets
Aug 06 20:38:41 dist-test-slave-tm1g krb5kdc[556](info): commencing operation
krb5kdc: starting...
W20260806 20:38:42.518736 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:38:42 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:38:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:38:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:38:42.749250 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46603
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:46603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:43.076378 569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:43.076658 569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:43.076831 569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:43.084437 569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:43.084546 569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:43.084672 569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:43.084774 569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:43.084846 569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:43.094696 569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:46603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:43.096040 569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:43.097757 569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:43.110498 574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:43.110378 577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:43.110477 575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:43.111294 569 server_base.cc:1034] running on GCE node
I20260806 20:38:43.112057 569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:43.113376 569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:43.114610 569 hybrid_clock.cc:648] HybridClock initialized: now 1786048723114570 us; error 50 us; skew 500 ppm
I20260806 20:38:43.114959 569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:43 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048723, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:43.118086 569 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:38:43.119683 569 webserver.cc:460] Webserver started at http://127.23.200.62:35647/ using document root <none> and password file <none>
I20260806 20:38:43.120396 569 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:43.120548 569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:43.120870 569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:43.123898 569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "64a59c976b2a4c50a4943d31f52ffce8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "02cbdba309870f8d0b5bbf19b062c614"
server_key_iv: "32daef29e5b9bb323e20f0ca2fa910cf"
server_key_version: "encryptionkey@0"
I20260806 20:38:43.124661 569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "64a59c976b2a4c50a4943d31f52ffce8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "02cbdba309870f8d0b5bbf19b062c614"
server_key_iv: "32daef29e5b9bb323e20f0ca2fa910cf"
server_key_version: "encryptionkey@0"
I20260806 20:38:43.129806 569 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:38:43.133153 584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:43.134429 569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:43.134622 569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "64a59c976b2a4c50a4943d31f52ffce8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "02cbdba309870f8d0b5bbf19b062c614"
server_key_iv: "32daef29e5b9bb323e20f0ca2fa910cf"
server_key_version: "encryptionkey@0"
I20260806 20:38:43.134822 569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:43.153631 569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:43.157886 569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:43.158213 569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:43.177197 569 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:46603
I20260806 20:38:43.177219 637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:46603 every 8 connection(s)
I20260806 20:38:43.178606 569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:43.181351 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 569
I20260806 20:38:43.181614 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:38:43.182471 24352 external_mini_cluster.cc:1428] Setting key 28e1f18923ad25a7217195339a48ec3e
I20260806 20:38:43.182940 638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:38:43.197535 638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Bootstrap starting.
Aug 06 20:38:43 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:43.201233 638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Neither blocks nor log segments found. Creating new log.
I20260806 20:38:43.202288 638 log.cc:824] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:43.205583 638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: No bootstrap required, opened a new log
I20260806 20:38:43.206836 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:43.184505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.23.200.62:46603)
0806 20:38:43.185274 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:43.185285 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:43.185336 (+ 51us) server_negotiation.cc:407] Connection header received
0806 20:38:43.187629 (+ 2293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:43.187649 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:43.188023 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:43.188341 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:43.190260 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:43.191640 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:43.193524 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:43.194234 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:43.194503 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:43.199380 (+ 4877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:43.199388 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:43.199400 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:43.199443 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:43.202672 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:43.203450 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:43.203459 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:43.203466 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:43.203686 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:43.204095 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:43.204098 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:43.204100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:43.204597 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:43.204808 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:43.205336 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 20:38:43.205699 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":378,"threads_started":1}
I20260806 20:38:43.213069 638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:43.213394 638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:38:43.213457 638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a59c976b2a4c50a4943d31f52ffce8, State: Initialized, Role: FOLLOWER
I20260806 20:38:43.214030 638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:43.214255 638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:43.214423 638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:43.214557 638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:38:43.217053 638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:43.217489 638 leader_election.cc:302] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a59c976b2a4c50a4943d31f52ffce8; no voters:
I20260806 20:38:43.217983 638 leader_election.cc:288] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:38:43.218142 643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:38:43.218828 643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 LEADER]: Becoming Leader. State: Replica: 64a59c976b2a4c50a4943d31f52ffce8, State: Running, Role: LEADER
I20260806 20:38:43.219395 643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:43.220060 638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:43.222033 644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a59c976b2a4c50a4943d31f52ffce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } } }
I20260806 20:38:43.222393 644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:43.223233 648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:43.223129 645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a59c976b2a4c50a4943d31f52ffce8. Latest consensus state: current_term: 1 leader_uuid: "64a59c976b2a4c50a4943d31f52ffce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } } }
I20260806 20:38:43.223475 645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:43.229362 648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:43.241452 648 catalog_manager.cc:1347] Generated new cluster ID: 7ccd2e6d02074d28a5f5ad4254e09627
I20260806 20:38:43.241580 648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:43.252511 648 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:38:43.254240 648 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:43.260489 648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Generated new TSK 0
I20260806 20:38:43.261286 648 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:38:43.318760 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:46603
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:43.651258 666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:43.651536 666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:43.651633 666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:43.658670 666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:43.658771 666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:43.658954 666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:43.668452 666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:43.669688 666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:43.671008 666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:43.683969 674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:43.684027 672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:43.684053 671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:43.684743 666 server_base.cc:1034] running on GCE node
I20260806 20:38:43.685345 666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:43.686201 666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:43.687484 666 hybrid_clock.cc:648] HybridClock initialized: now 1786048723687445 us; error 58 us; skew 500 ppm
I20260806 20:38:43.687760 666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:43 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048723, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:43.690750 666 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:38:43.692206 666 webserver.cc:460] Webserver started at http://127.23.200.1:40769/ using document root <none> and password file <none>
I20260806 20:38:43.692890 666 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:43.693040 666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:43.693349 666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:43.696237 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "5f5985da3457480c9fa5742fef6d1aa8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "1565f61b4adaabff6d53854db2cb57fe"
server_key_iv: "bfb9de536131ed01989fa301ad202b4e"
server_key_version: "encryptionkey@0"
I20260806 20:38:43.697021 666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "5f5985da3457480c9fa5742fef6d1aa8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "1565f61b4adaabff6d53854db2cb57fe"
server_key_iv: "bfb9de536131ed01989fa301ad202b4e"
server_key_version: "encryptionkey@0"
I20260806 20:38:43.702109 666 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 20:38:43.705585 682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:43.707226 666 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:43.707404 666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "5f5985da3457480c9fa5742fef6d1aa8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "1565f61b4adaabff6d53854db2cb57fe"
server_key_iv: "bfb9de536131ed01989fa301ad202b4e"
server_key_version: "encryptionkey@0"
I20260806 20:38:43.707652 666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:43.728569 666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:43.732733 666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:43.732978 666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:43.733855 666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:43.735249 666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:43.735327 666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:43.735407 666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:43.735451 666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:43.774590 666 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:36507
I20260806 20:38:43.774636 797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:36507 every 8 connection(s)
I20260806 20:38:43.775907 666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:43.780566 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 666
I20260806 20:38:43.780802 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:38:43.781445 24352 external_mini_cluster.cc:1428] Setting key 3f4fdc3160f081d54779af6798e17dd4
Aug 06 20:38:43 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048723, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:43.795902 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:43.778637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:54833 (local address 127.23.200.62:46603)
0806 20:38:43.779143 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:43.779148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:43.779856 (+ 708us) server_negotiation.cc:407] Connection header received
0806 20:38:43.781782 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:43.781788 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:43.781897 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:43.782018 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:43.783847 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:43.784556 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:43.786076 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:43.786546 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:43.786682 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:43.790278 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:43.790283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:43.790286 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:43.790363 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:43.792845 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:43.793580 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:43.793587 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:43.793591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:43.793717 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:43.794135 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:43.794141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:43.794144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:43.794464 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:43.794618 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:43.795292 (+ 674us) server_negotiation.cc:299] Negotiation successful
0806 20:38:43.795522 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":277,"threads_started":1}
I20260806 20:38:43.796947 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:43.778990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:54833)
0806 20:38:43.779695 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:43.779732 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:43.781482 (+ 1750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:43.782252 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:43.782260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:43.783064 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:43.783677 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:43.783699 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:43.784705 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:43.784711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:43.785937 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:43.785949 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:43.786792 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:43.786799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:43.787031 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:43.787715 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:43.787737 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:43.790067 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:43.793004 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:43.793013 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:43.793028 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:43.793462 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:43.793830 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:43.793835 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:43.793838 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:43.794023 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:43.794589 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:43.794597 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:43.794932 (+ 335us) client_negotiation.cc:769] Sending connection context
0806 20:38:43.795196 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 20:38:43.795746 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":264,"threads_started":1}
I20260806 20:38:43.798636 798 heartbeater.cc:347] Connected to a master server at 127.23.200.62:46603
I20260806 20:38:43.798954 798 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:43.799641 798 heartbeater.cc:511] Master 127.23.200.62:46603 requested a full tablet report, sending...
I20260806 20:38:43.802011 602 ts_manager.cc:194] Registered new tserver with Master: 5f5985da3457480c9fa5742fef6d1aa8 (127.23.200.1:36507)
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:38:43.804140 602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:54833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:38:43.831892 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:46603
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:44.161705 804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:44.161965 804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:44.162122 804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:44.169180 804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:44.169281 804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:44.169389 804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:44.178829 804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:44.180080 804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:44.181394 804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:44.194456 810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:44.194501 809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:44.194401 812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:44.195483 804 server_base.cc:1034] running on GCE node
I20260806 20:38:44.196097 804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:44.197023 804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:44.198247 804 hybrid_clock.cc:648] HybridClock initialized: now 1786048724198194 us; error 49 us; skew 500 ppm
I20260806 20:38:44.198575 804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:44 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:44.201531 804 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:38:44.202944 804 webserver.cc:460] Webserver started at http://127.23.200.2:38775/ using document root <none> and password file <none>
I20260806 20:38:44.203572 804 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:44.203672 804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:44.203907 804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:44.206807 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "046fc7a6107d44eb96e0407cedbf3c80"
format_stamp: "Formatted at 2026-08-06 20:38:44 on dist-test-slave-tm1g"
server_key: "e799d1fc7accca8aaf3dd762b6b1faea"
server_key_iv: "2c21ad2de8dd3bb5199c941ad8002bd3"
server_key_version: "encryptionkey@0"
I20260806 20:38:44.207530 804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "046fc7a6107d44eb96e0407cedbf3c80"
format_stamp: "Formatted at 2026-08-06 20:38:44 on dist-test-slave-tm1g"
server_key: "e799d1fc7accca8aaf3dd762b6b1faea"
server_key_iv: "2c21ad2de8dd3bb5199c941ad8002bd3"
server_key_version: "encryptionkey@0"
I20260806 20:38:44.212623 804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:38:44.215840 820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:44.217070 804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:38:44.217247 804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "046fc7a6107d44eb96e0407cedbf3c80"
format_stamp: "Formatted at 2026-08-06 20:38:44 on dist-test-slave-tm1g"
server_key: "e799d1fc7accca8aaf3dd762b6b1faea"
server_key_iv: "2c21ad2de8dd3bb5199c941ad8002bd3"
server_key_version: "encryptionkey@0"
I20260806 20:38:44.217437 804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:44.232018 804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:44.235919 804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:44.236208 804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:44.237110 804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:44.238533 804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:44.238672 804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:44.238818 804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:44.238894 804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:44.272406 804 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:46485
I20260806 20:38:44.272454 938 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:46485 every 8 connection(s)
I20260806 20:38:44.273686 804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:44.283785 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 804
I20260806 20:38:44.284000 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:38:44.284472 24352 external_mini_cluster.cc:1428] Setting key cdb3fbd650e6e0a08517fd489c9bd0c0
Aug 06 20:38:44 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048724, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:44.295032 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:44.276193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:33441 (local address 127.23.200.62:46603)
0806 20:38:44.276382 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:44.276386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:44.277549 (+ 1163us) server_negotiation.cc:407] Connection header received
0806 20:38:44.279384 (+ 1835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:44.279389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:44.279457 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:44.279552 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:44.280978 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:44.281690 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:44.283222 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:44.283841 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:44.283982 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:44.288113 (+ 4131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:44.288121 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:44.288129 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:44.288177 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:44.291745 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:44.292473 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:44.292481 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:44.292488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:44.292651 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:44.293042 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:44.293049 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:44.293055 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:44.293442 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:44.293601 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:44.294284 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 20:38:44.294582 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:38:44.295737 941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:44.276671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:33441)
0806 20:38:44.277360 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:44.277403 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:44.279138 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:44.279755 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:44.279763 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:44.280266 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:44.280831 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:44.280849 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:44.281819 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:44.281823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:44.283099 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:44.283111 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:44.284092 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:44.284101 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:44.284329 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:44.285058 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:44.285088 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:44.287875 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:44.291886 (+ 4011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:44.291894 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:44.291907 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:44.292333 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:44.292757 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:44.292761 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:44.292763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:44.292936 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:44.293565 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:44.293571 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:44.293890 (+ 319us) client_negotiation.cc:769] Sending connection context
0806 20:38:44.294120 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 20:38:44.294590 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":268,"threads_started":1}
I20260806 20:38:44.297406 939 heartbeater.cc:347] Connected to a master server at 127.23.200.62:46603
I20260806 20:38:44.297710 939 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:44.298492 939 heartbeater.cc:511] Master 127.23.200.62:46603 requested a full tablet report, sending...
I20260806 20:38:44.300014 602 ts_manager.cc:194] Registered new tserver with Master: 046fc7a6107d44eb96e0407cedbf3c80 (127.23.200.2:46485)
I20260806 20:38:44.301321 602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:33441
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:38:44.340883 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:46603
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:38:44.807683 798 heartbeater.cc:503] Master 127.23.200.62:46603 was elected leader, sending a full tablet report...
W20260806 20:38:45.128360 944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:45.128679 944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:45.128857 944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:45.135998 944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:45.136143 944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:45.136332 944 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:45.146027 944 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:45.147336 944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:45.148661 944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:45.161836 950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:45.161762 953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:45.161815 951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:45.162899 944 server_base.cc:1034] running on GCE node
I20260806 20:38:45.163528 944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:45.164377 944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:45.165580 944 hybrid_clock.cc:648] HybridClock initialized: now 1786048725165524 us; error 49 us; skew 500 ppm
I20260806 20:38:45.165889 944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:45 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:45.169106 944 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:38:45.170674 944 webserver.cc:460] Webserver started at http://127.23.200.3:40659/ using document root <none> and password file <none>
I20260806 20:38:45.171352 944 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:45.171480 944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:45.171801 944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:38:45.174692 944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "2b9c3a4cef0547fcba3b3cecefdaefa2"
format_stamp: "Formatted at 2026-08-06 20:38:45 on dist-test-slave-tm1g"
server_key: "3121f3e5532946f0a52fcae967d6a6d7"
server_key_iv: "19bb54f10841810479ceec088694da40"
server_key_version: "encryptionkey@0"
I20260806 20:38:45.175468 944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "2b9c3a4cef0547fcba3b3cecefdaefa2"
format_stamp: "Formatted at 2026-08-06 20:38:45 on dist-test-slave-tm1g"
server_key: "3121f3e5532946f0a52fcae967d6a6d7"
server_key_iv: "19bb54f10841810479ceec088694da40"
server_key_version: "encryptionkey@0"
I20260806 20:38:45.180696 944 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:38:45.184523 962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:45.185832 944 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 20:38:45.186005 944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "2b9c3a4cef0547fcba3b3cecefdaefa2"
format_stamp: "Formatted at 2026-08-06 20:38:45 on dist-test-slave-tm1g"
server_key: "3121f3e5532946f0a52fcae967d6a6d7"
server_key_iv: "19bb54f10841810479ceec088694da40"
server_key_version: "encryptionkey@0"
I20260806 20:38:45.186198 944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:45.198096 944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:45.202246 944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:45.202558 944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:45.203490 944 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:45.204792 944 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:45.204900 944 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:45.205041 944 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:45.205116 944 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:45.248514 944 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:33383
I20260806 20:38:45.248561 1078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:33383 every 8 connection(s)
I20260806 20:38:45.249838 944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:45.258285 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 944
I20260806 20:38:45.258564 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:38:45.259047 24352 external_mini_cluster.cc:1428] Setting key 1b0bd9cf79036cda8f05e0c34dfc8cfd
Aug 06 20:38:45 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048725, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:38:45.269440 1081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:45.252406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:35889 (local address 127.23.200.62:46603)
0806 20:38:45.252954 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:45.252960 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:45.253786 (+ 826us) server_negotiation.cc:407] Connection header received
0806 20:38:45.255751 (+ 1965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:45.255755 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:45.255822 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:45.255920 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:45.257500 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.258392 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.260042 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.260473 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.260609 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:45.264109 (+ 3500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:45.264113 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:45.264117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:45.264148 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:45.266860 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.267519 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.267523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.267525 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.267636 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.267995 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.267998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.268000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.268243 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:45.268358 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:45.268984 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 20:38:45.269146 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":249,"threads_started":1}
I20260806 20:38:45.270282 1082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:45.252836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:35889)
0806 20:38:45.253624 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:45.253668 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:45.255498 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:45.256133 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:45.256142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:45.256726 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:45.257319 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:45.257336 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.258568 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.258572 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:45.259891 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:45.259907 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.260694 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.260702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:45.260933 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:45.261599 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:45.261620 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:45.263907 (+ 2287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:38:45.266980 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:45.266987 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:45.266997 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:45.267413 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:45.267728 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:38:45.267731 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:38:45.267733 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:38:45.267906 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:38:45.268344 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:38:45.268349 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:38:45.268627 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 20:38:45.268854 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 20:38:45.269221 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":304,"threads_started":1}
I20260806 20:38:45.271703 1079 heartbeater.cc:347] Connected to a master server at 127.23.200.62:46603
I20260806 20:38:45.272001 1079 heartbeater.cc:464] Registering TS with master...
I20260806 20:38:45.272738 1079 heartbeater.cc:511] Master 127.23.200.62:46603 requested a full tablet report, sending...
I20260806 20:38:45.274428 602 ts_manager.cc:194] Registered new tserver with Master: 2b9c3a4cef0547fcba3b3cecefdaefa2 (127.23.200.3:33383)
I20260806 20:38:45.275147 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:38:45.275256 602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:35889
WARNING: no policy specified for oryx/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.1@KRBTEST.COM" created.
I20260806 20:38:45.304423 939 heartbeater.cc:503] Master 127.23.200.62:46603 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab.
Entry for principal oryx/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab.
WARNING: no policy specified for oryx/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab.
Entry for principal oryx/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab.
WARNING: no policy specified for oryx/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab.
Entry for principal oryx/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab.
I20260806 20:38:45.435333 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 666
Aug 06 20:38:45 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:38:45.452440 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:45.436664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58500 (local address 127.23.200.1:36507)
0806 20:38:45.437451 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:45.437458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:45.437492 (+ 34us) server_negotiation.cc:407] Connection header received
0806 20:38:45.437555 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:45.437587 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:45.437838 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:45.438027 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:45.439548 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.440646 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.441966 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.442657 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.442765 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:45.447086 (+ 4321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:45.447104 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:45.447120 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:45.447156 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:45.449922 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.450636 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.450643 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.450646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.450772 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.451107 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.451110 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.451112 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.451449 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:45.451681 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:45.451953 (+ 272us) server_negotiation.cc:299] Negotiation successful
0806 20:38:45.452150 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":443,"threads_started":1}
I20260806 20:38:45.453811 793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:45.540272 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 666
I20260806 20:38:45.559489 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 804
Aug 06 20:38:45 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:38:45.576109 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:45.560660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.23.200.2:46485)
0806 20:38:45.561248 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:45.561255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:45.561281 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:38:45.561343 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:45.561349 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:45.561611 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:45.561741 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:45.563116 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.564411 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.565689 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.566249 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.566425 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:45.570586 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:45.570598 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:45.570613 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:45.570649 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:45.573547 (+ 2898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.574172 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.574179 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.574182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.574478 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.574823 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.574826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.574828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.575152 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:45.575370 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:45.575619 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 20:38:45.575817 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":280,"threads_started":1}
I20260806 20:38:45.577386 934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:45.662110 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 804
I20260806 20:38:45.682087 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 944
Aug 06 20:38:45 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:38:45.702499 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:45.683393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55984 (local address 127.23.200.3:33383)
0806 20:38:45.684332 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:45.684349 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:45.684381 (+ 32us) server_negotiation.cc:407] Connection header received
0806 20:38:45.684455 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:45.684462 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:45.684768 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:45.684921 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:45.686480 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.687549 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.689000 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:45.689740 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:45.689871 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:45.695336 (+ 5465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:45.695354 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:45.695372 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:45.695420 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:45.698891 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.699784 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.699793 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.699796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.699976 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:45.700492 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:45.700500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:45.700502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:45.700922 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:45.701183 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:45.701613 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 20:38:45.701872 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":448,"threads_started":1}
I20260806 20:38:45.704131 1073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:45.799477 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 944
I20260806 20:38:45.818397 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 569
I20260806 20:38:45.819955 633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:38:45.927783 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 569
I20260806 20:38:45.949959 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46603
--webserver_interface=127.23.200.62
--webserver_port=35647
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:46603
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:38:46.270516 1098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:46.270782 1098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:46.270872 1098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:46.277807 1098 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:38:46.277907 1098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:46.277951 1098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:46.278009 1098 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:38:46.278052 1098 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:38:46.287462 1098 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:46603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=35647
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:46.288677 1098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:46.289906 1098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:46.301723 1104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:46.301704 1106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:46.302040 1103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:46.302898 1098 server_base.cc:1034] running on GCE node
I20260806 20:38:46.303699 1098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:46.305075 1098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:46.306362 1098 hybrid_clock.cc:648] HybridClock initialized: now 1786048726306248 us; error 71 us; skew 500 ppm
I20260806 20:38:46.306726 1098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:46 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048726, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:46.310096 1098 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:38:46.312541 1098 webserver.cc:460] Webserver started at http://127.23.200.62:35647/ using document root <none> and password file <none>
I20260806 20:38:46.313252 1098 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:46.313412 1098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:46.318861 1098 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:38:46.321590 1113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:46.323033 1098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:46.323208 1098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "64a59c976b2a4c50a4943d31f52ffce8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "02cbdba309870f8d0b5bbf19b062c614"
server_key_iv: "32daef29e5b9bb323e20f0ca2fa910cf"
server_key_version: "encryptionkey@0"
I20260806 20:38:46.324177 1098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:46.344051 1098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:46.352123 1098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:46.352406 1098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:46.370782 1098 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:46603
I20260806 20:38:46.370815 1167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:46603 every 8 connection(s)
I20260806 20:38:46.372285 1098 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:38:46.378865 1168 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:38:46.381938 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1098
I20260806 20:38:46.382267 1168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Bootstrap starting.
I20260806 20:38:46.382961 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:36507
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:46603
--webserver_port=40769
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:38:46.404119 1168 log.cc:824] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Log is configured to *not* fsync() on all Append() calls
I20260806 20:38:46.412600 1168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:38:46.413122 1168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Bootstrap complete.
I20260806 20:38:46.422102 1168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:46.422981 1168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a59c976b2a4c50a4943d31f52ffce8, State: Initialized, Role: FOLLOWER
I20260806 20:38:46.423645 1168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:46.423879 1168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:38:46.424019 1168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:38:46.424194 1168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:38:46.428061 1168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:46.428552 1168 leader_election.cc:302] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a59c976b2a4c50a4943d31f52ffce8; no voters:
I20260806 20:38:46.429237 1168 leader_election.cc:288] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 20:38:46.429391 1172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:38:46.431635 1168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:38:46.436467 1172 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [term 2 LEADER]: Becoming Leader. State: Replica: 64a59c976b2a4c50a4943d31f52ffce8, State: Running, Role: LEADER
I20260806 20:38:46.437170 1172 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } }
I20260806 20:38:46.440153 1173 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "64a59c976b2a4c50a4943d31f52ffce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } } }
I20260806 20:38:46.440305 1174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a59c976b2a4c50a4943d31f52ffce8. Latest consensus state: current_term: 2 leader_uuid: "64a59c976b2a4c50a4943d31f52ffce8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a59c976b2a4c50a4943d31f52ffce8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46603 } } }
I20260806 20:38:46.440610 1173 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:46.440632 1174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8 [sys.catalog]: This master's current role is: LEADER
I20260806 20:38:46.442842 1184 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:38:46.454764 1184 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:38:46.456200 1184 catalog_manager.cc:1259] Loaded cluster ID: 7ccd2e6d02074d28a5f5ad4254e09627
I20260806 20:38:46.456348 1184 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:38:46.460371 1184 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:38:46.461814 1184 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 64a59c976b2a4c50a4943d31f52ffce8: Loaded TSK: 0
I20260806 20:38:46.463366 1184 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 20:38:46.728550 1170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:46.728833 1170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:46.728987 1170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:46.736699 1170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:46.736807 1170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:46.736991 1170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:38:46.747277 1170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.1.keytab
--principal=oryx/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:36507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=40769
--tserver_master_addrs=127.23.200.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:46.748637 1170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:46.749969 1170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:46.763468 1197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:46.763447 1196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:46.768685 1199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:46.769644 1170 server_base.cc:1034] running on GCE node
I20260806 20:38:46.770458 1170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:46.771631 1170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:46.772881 1170 hybrid_clock.cc:648] HybridClock initialized: now 1786048726772806 us; error 63 us; skew 500 ppm
I20260806 20:38:46.773240 1170 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:46 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048726, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:46.776705 1170 init.cc:377] Logged in from keytab as oryx/127.23.200.1@KRBTEST.COM (short username oryx)
I20260806 20:38:46.778458 1170 webserver.cc:460] Webserver started at http://127.23.200.1:40769/ using document root <none> and password file <none>
I20260806 20:38:46.779225 1170 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:46.779384 1170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:46.784929 1170 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:38:46.787858 1206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:46.789266 1170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:38:46.789444 1170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "5f5985da3457480c9fa5742fef6d1aa8"
format_stamp: "Formatted at 2026-08-06 20:38:43 on dist-test-slave-tm1g"
server_key: "1565f61b4adaabff6d53854db2cb57fe"
server_key_iv: "bfb9de536131ed01989fa301ad202b4e"
server_key_version: "encryptionkey@0"
I20260806 20:38:46.790442 1170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:46.814030 1170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:46.818696 1170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:46.818989 1170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:46.820024 1170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:46.821498 1170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:46.821589 1170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:46.821740 1170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:46.821836 1170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:46.856936 1170 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:36507
I20260806 20:38:46.857033 1325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:36507 every 8 connection(s)
I20260806 20:38:46.858486 1170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:38:46.868363 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1170
I20260806 20:38:46.868989 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:46485
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:46603
--webserver_port=38775
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:38:46 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:46 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 20:38:47.207921 1330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:47.208195 1330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:47.208357 1330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:47.215621 1330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:47.215750 1330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:47.215934 1330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:38:47.225559 1330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.2.keytab
--principal=oryx/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:46485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=38775
--tserver_master_addrs=127.23.200.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:47.226955 1330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:47.228345 1330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:47.247571 1336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:47.247571 1337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:47.247869 1339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:47.249402 1330 server_base.cc:1034] running on GCE node
I20260806 20:38:47.250511 1330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:47.252287 1330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:47.253641 1330 hybrid_clock.cc:648] HybridClock initialized: now 1786048727253551 us; error 56 us; skew 500 ppm
I20260806 20:38:47.254211 1330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:47 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:47.259891 1330 init.cc:377] Logged in from keytab as oryx/127.23.200.2@KRBTEST.COM (short username oryx)
I20260806 20:38:47.262328 1330 webserver.cc:460] Webserver started at http://127.23.200.2:38775/ using document root <none> and password file <none>
I20260806 20:38:47.263679 1330 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:47.263895 1330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:47.274055 1330 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 20:38:47.278800 1346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:47.281033 1330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:38:47.281294 1330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "046fc7a6107d44eb96e0407cedbf3c80"
format_stamp: "Formatted at 2026-08-06 20:38:44 on dist-test-slave-tm1g"
server_key: "e799d1fc7accca8aaf3dd762b6b1faea"
server_key_iv: "2c21ad2de8dd3bb5199c941ad8002bd3"
server_key_version: "encryptionkey@0"
I20260806 20:38:47.283383 1330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:47.329145 1330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:47.335314 1330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:47.335609 1330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:47.336702 1330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:47.338358 1330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:47.338632 1330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 20:38:47.338775 1330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:47.338881 1330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:47.379179 1330 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:46485
I20260806 20:38:47.379184 1464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:46485 every 8 connection(s)
I20260806 20:38:47.380654 1330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:38:47.383939 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1330
I20260806 20:38:47.384629 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:33383
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:46603
--webserver_port=40659
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:38:47 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:47 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
W20260806 20:38:47.729414 1469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:38:47.729709 1469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:38:47.729880 1469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:38:47.737259 1469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:38:47.737407 1469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:38:47.737591 1469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:38:47.753067 1469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/krb5kdc/oryx_127.23.200.3.keytab
--principal=oryx/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:33383
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=40659
--tserver_master_addrs=127.23.200.62:46603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:38:47.755553 1469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:38:47.757377 1469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:38:47.794138 1476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:47.802116 1475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:38:47.803752 1478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:38:47.803920 1469 server_base.cc:1034] running on GCE node
I20260806 20:38:47.804615 1469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:38:47.806587 1469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:38:47.807814 1469 hybrid_clock.cc:648] HybridClock initialized: now 1786048727807746 us; error 57 us; skew 500 ppm
I20260806 20:38:47.808143 1469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:38:47 dist-test-slave-tm1g krb5kdc[556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048727, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:38:47.811635 1469 init.cc:377] Logged in from keytab as oryx/127.23.200.3@KRBTEST.COM (short username oryx)
I20260806 20:38:47.813537 1469 webserver.cc:460] Webserver started at http://127.23.200.3:40659/ using document root <none> and password file <none>
I20260806 20:38:47.814258 1469 fs_manager.cc:359] Metadata directory not provided
I20260806 20:38:47.814436 1469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:38:47.819839 1469 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260806 20:38:47.822594 1485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:47.824003 1469 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 20:38:47.824175 1469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "2b9c3a4cef0547fcba3b3cecefdaefa2"
format_stamp: "Formatted at 2026-08-06 20:38:45 on dist-test-slave-tm1g"
server_key: "3121f3e5532946f0a52fcae967d6a6d7"
server_key_iv: "19bb54f10841810479ceec088694da40"
server_key_version: "encryptionkey@0"
I20260806 20:38:47.825111 1469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:38:47.848564 1469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:38:47.853019 1469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:38:47.853292 1469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:38:47.854349 1469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:38:47.855774 1469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:38:47.855893 1469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:47.856042 1469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:38:47.856146 1469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:38:47.913331 1469 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:33383
I20260806 20:38:47.913380 1603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:33383 every 8 connection(s)
I20260806 20:38:47.915001 1469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:38:47.922441 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1469
Aug 06 20:38:47 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:47 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:47.950745 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:47.923968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.23.200.62:46603)
0806 20:38:47.926832 (+ 2864us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:47.926839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:47.926859 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:38:47.926913 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:47.926919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:47.926987 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:47.927088 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:47.930038 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.930990 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.933204 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.933811 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.933941 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:47.939806 (+ 5865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:38:47.939822 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:38:47.939849 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:38:47.940331 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:38:47.944647 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:47.945594 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:47.945607 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:47.945619 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:47.945835 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:38:47.946321 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:38:47.946329 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:38:47.946335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:38:47.947020 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:38:47.947335 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 20:38:47.948914 (+ 1579us) server_negotiation.cc:299] Negotiation successful
0806 20:38:47.949428 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2733,"thread_start_us":2618,"threads_started":1}
W20260806 20:38:48.423354 1329 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.551s user 0.005s sys 0.010s
W20260806 20:38:48.425590 1328 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:46.861580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:40179 (local address 127.23.200.62:46603)
0806 20:38:46.862515 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:46.862534 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:46.862774 (+ 240us) server_negotiation.cc:407] Connection header received
0806 20:38:46.864995 (+ 2221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:46.865009 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:46.865370 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:46.865737 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:46.867536 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:46.868859 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:46.870603 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:46.871277 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:46.871515 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.425086 (+1553571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:40179: BlockingRecv error: recv got EOF from 127.23.200.1:40179 (error 108)
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":347,"threads_started":1}
I20260806 20:38:48.427982 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:46.861679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:40179)
0806 20:38:46.862594 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:46.862647 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:46.864424 (+ 1777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:46.865901 (+ 1477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:46.865911 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:46.866772 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:46.867324 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:46.867342 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:46.869023 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:46.869028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:46.870458 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:46.870471 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:46.871558 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:46.871566 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:46.871863 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:46.872571 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:46.872594 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:48.426203 (+1553609us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":360,"threads_started":1}
W20260806 20:38:48.429598 1326 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:46603 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.200.62:46603: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:48.446746 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:48.430608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:55363)
0806 20:38:48.431182 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:48.431211 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:48.431409 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:48.431895 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:48.431903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:48.432624 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:48.432979 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.432997 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.435199 (+ 2202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.435208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.437534 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.437563 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.439087 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.439093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.439285 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.440240 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:48.440278 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:48.446239 (+ 5961us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381}
W20260806 20:38:48.449479 1328 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:48.430680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:55363 (local address 127.23.200.62:46603)
0806 20:38:48.430944 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:48.430954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:48.431330 (+ 376us) server_negotiation.cc:407] Connection header received
0806 20:38:48.431576 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:48.431585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:48.431698 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:48.431849 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:48.433181 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.434935 (+ 1754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.437756 (+ 2821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.438820 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.439026 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.446615 (+ 7589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:55363: BlockingRecv error: recv got EOF from 127.23.200.1:55363 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:48.465415 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:48.447900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:48469)
0806 20:38:48.450629 (+ 2729us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:48.450658 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:48.450822 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:48.451412 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:48.451416 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:48.451693 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:48.451839 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.451846 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.453793 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.453802 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.456207 (+ 2405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.456229 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.457676 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.457685 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.457995 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.459013 (+ 1018us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:48.459049 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:48.464936 (+ 5887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2581,"thread_start_us":3181,"threads_started":1}
W20260806 20:38:48.465757 1610 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:48.448445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:48469 (local address 127.23.200.62:46603)
0806 20:38:48.448690 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:48.448703 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:48.450747 (+ 2044us) server_negotiation.cc:407] Connection header received
0806 20:38:48.451012 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:48.451020 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:48.451123 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:48.451228 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:48.451991 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.453591 (+ 1600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.456426 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.457457 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.457642 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.465339 (+ 7697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:48469: BlockingRecv error: recv got EOF from 127.23.200.1:48469 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 20:38:48.466181 1326 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 20:38:48.858259 1467 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.449s user 0.000s sys 0.018s
W20260806 20:38:48.859966 1468 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:47.385168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:45579 (local address 127.23.200.62:46603)
0806 20:38:47.393732 (+ 8564us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:47.393738 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:47.393759 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:38:47.393822 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:47.393830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:47.393915 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:47.394032 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:47.401301 (+ 7269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.402667 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.406092 (+ 3425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.406918 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.407071 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.859716 (+1452645us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:45579: BlockingRecv error: recv got EOF from 127.23.200.2:45579 (error 108)
Metrics: {"server-negotiator.queue_time_us":8404,"thread_start_us":8305,"threads_started":1}
I20260806 20:38:48.861305 1467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:47.384490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:45579)
0806 20:38:47.390874 (+ 6384us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:47.390922 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:47.393082 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:47.398799 (+ 5717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:47.398818 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:47.399901 (+ 1083us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:47.401029 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:47.401061 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.402895 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.402904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:47.405863 (+ 2959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:47.405903 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.407243 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.407253 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:47.407705 (+ 452us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:47.408759 (+ 1054us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:47.408804 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:48.859601 (+1450797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":6067,"thread_start_us":5841,"threads_started":1}
W20260806 20:38:48.862481 1465 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:46603 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.200.62:46603: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:48.875813 1467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:48.863342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:60793)
0806 20:38:48.863712 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:48.863742 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:48.863922 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:48.864282 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:48.864290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:48.864968 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:48.865306 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.865323 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.866393 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.866401 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.868742 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.868763 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.869511 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.869520 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.869794 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.870654 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:48.870690 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:48.875537 (+ 4847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260806 20:38:48.875933 1468 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:48.863483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:60793 (local address 127.23.200.62:46603)
0806 20:38:48.863658 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:48.863662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:48.863833 (+ 171us) server_negotiation.cc:407] Connection header received
0806 20:38:48.864049 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:48.864053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:48.864123 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:48.864247 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:48.865472 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.866205 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.868914 (+ 2709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.869291 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.869381 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.875710 (+ 6329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:60793: BlockingRecv error: recv got EOF from 127.23.200.2:60793 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:48 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:48.885558 1467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:48.876722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:52261)
0806 20:38:48.877100 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:48.877118 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:48.877240 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:48.877589 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:48.877593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:48.877901 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:48.878064 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.878072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.879279 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.879284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.880216 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:48.880228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.880974 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.880980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:48.881174 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.881748 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:48.881764 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:48.885368 (+ 3604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268}
W20260806 20:38:48.885859 1465 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 20:38:48.886142 1468 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:48.876906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:52261 (local address 127.23.200.62:46603)
0806 20:38:48.877066 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:48.877072 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:48.877186 (+ 114us) server_negotiation.cc:407] Connection header received
0806 20:38:48.877369 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:48.877376 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:48.877453 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:48.877553 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:48.878200 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.879134 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.880381 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:48.880810 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:48.880938 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:48.885752 (+ 4814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:52261: BlockingRecv error: recv got EOF from 127.23.200.2:52261 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 20:38:49.063740 1606 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.127s user 0.002s sys 0.015s
W20260806 20:38:49.065081 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:47.919778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:46517 (local address 127.23.200.62:46603)
0806 20:38:47.925971 (+ 6193us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:47.925980 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:47.926006 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:38:47.926086 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:47.926095 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:47.926196 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:47.929760 (+ 3564us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:47.931181 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.932381 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.934805 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.935481 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.935607 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.064764 (+1129157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:46517: BlockingRecv error: recv got EOF from 127.23.200.3:46517 (error 108)
Metrics: {"server-negotiator.queue_time_us":6040,"thread_start_us":5917,"threads_started":1}
I20260806 20:38:49.065793 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:47.919074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:46517)
0806 20:38:47.921517 (+ 2443us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:47.921565 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:47.923980 (+ 2415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:47.927654 (+ 3674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:47.927667 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:47.928800 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:47.929943 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:47.929972 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.932555 (+ 2583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.932563 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:47.934631 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:47.934648 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:47.935944 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:47.935953 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:47.936299 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:47.937077 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:47.937119 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:49.064596 (+1127477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2121,"thread_start_us":1862,"threads_started":1}
W20260806 20:38:49.066732 1604 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:46603 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.200.62:46603: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:49.075623 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:49.067365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:35037)
0806 20:38:49.067564 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:49.067591 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:49.067681 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:49.067947 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:49.067950 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:49.068278 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:49.068444 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.068452 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.069411 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.069414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.070417 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.070429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.071269 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.071274 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.071388 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.071986 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:49.072004 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:49.075415 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 20:38:49.075918 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:49.067491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:35037 (local address 127.23.200.62:46603)
0806 20:38:49.067654 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:49.067659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:49.067675 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:38:49.067761 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:49.067764 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:49.067831 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:49.067910 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:49.068574 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.069291 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.070558 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.071110 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.071215 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.075633 (+ 4418us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:35037: BlockingRecv error: recv got EOF from 127.23.200.3:35037 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:49.084558 1606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:49.076269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:59661)
0806 20:38:49.076423 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:49.076439 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:49.076561 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:49.076909 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:49.076914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:49.077267 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:49.077469 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.077481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.078784 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.078789 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.079720 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.079731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.080440 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.080446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.080638 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.081220 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:49.081237 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:49.084367 (+ 3130us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260806 20:38:49.084820 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:49.076361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:59661 (local address 127.23.200.62:46603)
0806 20:38:49.076604 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:49.076611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:49.076630 (+ 19us) server_negotiation.cc:407] Connection header received
0806 20:38:49.076690 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:49.076694 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:49.076759 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:49.076863 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:49.077613 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.078639 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.079857 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.080271 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.080425 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.084544 (+ 4119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:59661: BlockingRecv error: recv got EOF from 127.23.200.3:59661 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
W20260806 20:38:49.084822 1604 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:49.476434 1613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:49.466850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:49609)
0806 20:38:49.467394 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:49.467409 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:49.467570 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:49.468118 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:49.468123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:49.468697 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:49.468911 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.468925 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.470116 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.470123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.471160 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.471172 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.471966 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.471974 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.472173 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.472750 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:49.472767 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:49.476196 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":304,"threads_started":1}
W20260806 20:38:49.476605 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:49.466936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49609 (local address 127.23.200.62:46603)
0806 20:38:49.467180 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:49.467189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:49.467504 (+ 315us) server_negotiation.cc:407] Connection header received
0806 20:38:49.467713 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:49.467725 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:49.467789 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:49.467902 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:49.469093 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.469958 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.471309 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.471751 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.471903 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.476399 (+ 4496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:49609: BlockingRecv error: recv got EOF from 127.23.200.1:49609 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:49 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:49.896024 1614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:49.886429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:39455)
0806 20:38:49.887036 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:49.887059 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:49.887199 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:49.887572 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:49.887576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:49.888044 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:49.888196 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.888205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.889259 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.889265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.890284 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:49.890326 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.891105 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.891112 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:49.891289 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.891927 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:49.891945 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:49.895673 (+ 3728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":325,"threads_started":1}
W20260806 20:38:49.896138 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:49.886908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:39455 (local address 127.23.200.62:46603)
0806 20:38:49.887127 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:49.887132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:49.887151 (+ 19us) server_negotiation.cc:407] Connection header received
0806 20:38:49.887362 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:49.887367 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:49.887431 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:49.887542 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:49.888382 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.889116 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.890475 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:49.890898 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:49.891081 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:49.895893 (+ 4812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:39455: BlockingRecv error: recv got EOF from 127.23.200.2:39455 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Aug 06 20:38:50 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:50 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:50.094535 1615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:50.085555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:52023)
0806 20:38:50.086125 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:50.086146 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:50.086257 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:50.086587 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:50.086591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:50.087030 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:50.087180 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:50.087189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.088206 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.088210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:50.089190 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:50.089203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.089908 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.089912 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:50.090032 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:50.090687 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:50.090729 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:50.094227 (+ 3498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":324,"threads_started":1}
W20260806 20:38:50.094785 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:50.085640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:52023 (local address 127.23.200.62:46603)
0806 20:38:50.085821 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:50.085825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:50.086219 (+ 394us) server_negotiation.cc:407] Connection header received
0806 20:38:50.086382 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:50.086386 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:50.086460 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:50.086542 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:50.087322 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.088061 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.089324 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.089749 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.089843 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:50.094505 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:52023: BlockingRecv error: recv got EOF from 127.23.200.3:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 20:38:50 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:50 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:50.486935 1616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:50.477525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:48735)
0806 20:38:50.478071 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:50.478088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:50.478216 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:50.478558 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:50.478562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:50.478986 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:50.479160 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:50.479170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.480623 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.480627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:50.481554 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:50.481566 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.482237 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.482241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:50.482428 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:50.483056 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:50.483074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:50.486577 (+ 3503us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":301,"threads_started":1}
W20260806 20:38:50.487074 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:50.477587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:48735 (local address 127.23.200.62:46603)
0806 20:38:50.477795 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:50.477800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:50.478196 (+ 396us) server_negotiation.cc:407] Connection header received
0806 20:38:50.478347 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:50.478351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:50.478421 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:50.478511 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:50.479306 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.480490 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.481681 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.482107 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.482203 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:50.486798 (+ 4595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:48735: BlockingRecv error: recv got EOF from 127.23.200.1:48735 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 06 20:38:50 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:50 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:50.905979 1617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:50.897016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:51027)
0806 20:38:50.897583 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:50.897598 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:50.897785 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:50.898171 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:50.898176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:50.898687 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:50.898872 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:50.898881 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.899907 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.899913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:50.900841 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:50.900852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.901584 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.901590 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:50.901761 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:50.902420 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:50.902437 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:50.905737 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":356,"threads_started":1}
W20260806 20:38:50.906380 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:50.897090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:51027 (local address 127.23.200.62:46603)
0806 20:38:50.897357 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:50.897365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:50.897692 (+ 327us) server_negotiation.cc:407] Connection header received
0806 20:38:50.897948 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:50.897953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:50.898017 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:50.898133 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:50.899021 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.899754 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.900978 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:50.901415 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:50.901546 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:50.905962 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:51027: BlockingRecv error: recv got EOF from 127.23.200.2:51027 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 20:38:51 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:51 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:51.104825 1618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:51.095533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:48391)
0806 20:38:51.096092 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:51.096109 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:51.096271 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:51.096672 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:51.096676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:51.097116 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:51.097302 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:51.097311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.098421 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.098425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:51.099323 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:51.099334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.100042 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.100048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:51.100234 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:51.100929 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:51.100952 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:51.104469 (+ 3517us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":337,"threads_started":1}
W20260806 20:38:51.104904 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:51.095764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:48391 (local address 127.23.200.62:46603)
0806 20:38:51.096002 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:51.096009 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:51.096195 (+ 186us) server_negotiation.cc:407] Connection header received
0806 20:38:51.096439 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:51.096444 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:51.096530 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:51.096638 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:51.097461 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.098229 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.099479 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.099887 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.100027 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:51.104652 (+ 4625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:48391: BlockingRecv error: recv got EOF from 127.23.200.3:48391 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Aug 06 20:38:51 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:51 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:51.502193 1619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:51.488057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:51463)
0806 20:38:51.488560 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:51.488577 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:51.488695 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:51.488978 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:51.488981 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:51.489364 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:51.489540 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:51.489549 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.490737 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.490741 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:51.491870 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:51.491883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.492782 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.492789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:51.492986 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:51.498407 (+ 5421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:51.498425 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:51.501945 (+ 3520us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":256,"threads_started":1}
W20260806 20:38:51.502388 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:51.488157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:51463 (local address 127.23.200.62:46603)
0806 20:38:51.488413 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:51.488419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:51.488665 (+ 246us) server_negotiation.cc:407] Connection header received
0806 20:38:51.488778 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:51.488782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:51.488851 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:51.488928 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:51.489738 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.490596 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.492013 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.492635 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.492770 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:51.502153 (+ 9383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:51463: BlockingRecv error: recv got EOF from 127.23.200.1:51463 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 06 20:38:51 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:51 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:51.916749 1620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:51.907091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:48333)
0806 20:38:51.907737 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:51.907756 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:51.907928 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:51.908351 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:51.908356 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:51.908903 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:51.909279 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:51.909290 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.910404 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.910412 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:51.911331 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:51.911343 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.912082 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.912088 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:51.912278 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:51.912912 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:51.912931 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:51.916512 (+ 3581us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":307,"threads_started":1}
W20260806 20:38:51.916965 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:51.907200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:48333 (local address 127.23.200.62:46603)
0806 20:38:51.907403 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:51.907409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:51.907845 (+ 436us) server_negotiation.cc:407] Connection header received
0806 20:38:51.908061 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:51.908065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:51.908169 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:51.908275 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:51.909465 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.910221 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.911479 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:51.911924 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:51.912056 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:51.916733 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:48333: BlockingRecv error: recv got EOF from 127.23.200.2:48333 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 20:38:52 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:52 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:52.115067 1621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:52.105817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:60973)
0806 20:38:52.106445 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:52.106462 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:52.106578 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:52.106884 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:52.106887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:52.107256 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:52.107449 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:52.107460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.108487 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.108492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:52.109445 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:52.109458 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.110143 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.110147 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:52.110272 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:52.110914 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:52.110942 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:52.114754 (+ 3812us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":319,"threads_started":1}
W20260806 20:38:52.115296 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:52.105876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:60973 (local address 127.23.200.62:46603)
0806 20:38:52.106054 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:52.106058 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:52.106534 (+ 476us) server_negotiation.cc:407] Connection header received
0806 20:38:52.106673 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:52.106677 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:52.106757 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:52.106839 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:52.107582 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.108318 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.109582 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.110007 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.110105 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:52.115019 (+ 4914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:60973: BlockingRecv error: recv got EOF from 127.23.200.3:60973 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 20:38:52 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:52 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:52.512547 1622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:52.503280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:41813)
0806 20:38:52.503836 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:52.503853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:52.503967 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:52.504285 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:52.504288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:52.504732 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:52.504959 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:52.504970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.506067 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.506072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:52.507158 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:52.507170 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.507891 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.507897 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:52.508094 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:52.508761 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:52.508779 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:52.512169 (+ 3390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
W20260806 20:38:52.512648 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:52.503470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:41813 (local address 127.23.200.62:46603)
0806 20:38:52.503659 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:52.503663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:52.503935 (+ 272us) server_negotiation.cc:407] Connection header received
0806 20:38:52.504070 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:52.504074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:52.504138 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:52.504227 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:52.505134 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.505921 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.507315 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.507741 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.507881 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:52.512386 (+ 4505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:41813: BlockingRecv error: recv got EOF from 127.23.200.1:41813 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 20:38:52 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:52 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:52.927363 1623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:52.917752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:60159)
0806 20:38:52.918284 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:52.918327 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:52.918483 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:52.918973 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:52.918980 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:52.919533 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:52.919795 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:52.919810 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.920991 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.920995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:52.921996 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:52.922009 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.922801 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.922805 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:52.922936 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:52.923599 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:52.923616 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:52.927010 (+ 3394us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":313,"threads_started":1}
W20260806 20:38:52.927453 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:52.917908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:60159 (local address 127.23.200.62:46603)
0806 20:38:52.918083 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:52.918088 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:52.918449 (+ 361us) server_negotiation.cc:407] Connection header received
0806 20:38:52.918640 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:52.918643 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:52.918771 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:52.918901 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:52.919984 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.920860 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.922125 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:52.922660 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:52.922758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:52.927217 (+ 4459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:60159: BlockingRecv error: recv got EOF from 127.23.200.2:60159 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 20:38:53 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:53 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:53.125988 1624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:53.116309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:35145)
0806 20:38:53.116802 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:53.116827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:53.116951 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:53.117388 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:53.117392 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:53.117782 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:53.117955 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:53.117964 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.119134 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.119139 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:53.120269 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:53.120282 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.121025 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.121030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:53.121171 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:53.121864 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:53.121884 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:53.125713 (+ 3829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":272,"threads_started":1}
W20260806 20:38:53.126164 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:53.116366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:35145 (local address 127.23.200.62:46603)
0806 20:38:53.116603 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:53.116610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:53.116916 (+ 306us) server_negotiation.cc:407] Connection header received
0806 20:38:53.117124 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:53.117128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:53.117203 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:53.117290 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:53.118133 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.118978 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.120426 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.120819 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.120920 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:53.125930 (+ 5010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:35145: BlockingRecv error: recv got EOF from 127.23.200.3:35145 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 20:38:53 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:53 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:53.523160 1625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:53.513617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:38739)
0806 20:38:53.514200 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:53.514217 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:53.514428 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:53.514784 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:53.514788 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:53.515236 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:53.515395 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:53.515404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.516413 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.516421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:53.517410 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:53.517421 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.518112 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.518117 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:53.518366 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:53.519056 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:53.519090 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:53.522923 (+ 3833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":296,"threads_started":1}
W20260806 20:38:53.523449 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:53.513714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38739 (local address 127.23.200.62:46603)
0806 20:38:53.513964 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:53.513972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:53.514335 (+ 363us) server_negotiation.cc:407] Connection header received
0806 20:38:53.514561 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:53.514570 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:53.514634 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:53.514747 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:53.515536 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.516281 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.517560 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.517956 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.518086 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:53.523157 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:38739: BlockingRecv error: recv got EOF from 127.23.200.1:38739 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 20:38:53 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:53 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:53.937736 1626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:53.928418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:59319)
0806 20:38:53.929024 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:53.929040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:53.929213 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:53.929606 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:53.929610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:53.930079 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:53.930237 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:53.930246 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.931392 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.931398 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:53.932472 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:53.932484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.933232 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.933238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:53.933426 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:53.933995 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:53.934012 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:53.937390 (+ 3378us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":338,"threads_started":1}
W20260806 20:38:53.937884 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:53.928595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:59319 (local address 127.23.200.62:46603)
0806 20:38:53.928856 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:53.928863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:53.929123 (+ 260us) server_negotiation.cc:407] Connection header received
0806 20:38:53.929354 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:53.929360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:53.929472 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:53.929571 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:53.930424 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.931241 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.932606 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:53.933073 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:53.933206 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:53.937611 (+ 4405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:59319: BlockingRecv error: recv got EOF from 127.23.200.2:59319 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Aug 06 20:38:54 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:54 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:54.137225 1627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:54.127180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:46295)
0806 20:38:54.127741 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:54.127758 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:54.127924 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:54.128348 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:54.128354 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:54.128949 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:54.129120 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:54.129130 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.130340 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.130346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:54.131276 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:54.131288 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.132231 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.132238 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:54.132420 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:54.133092 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:54.133110 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:54.136990 (+ 3880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":317,"threads_started":1}
W20260806 20:38:54.137562 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:54.127266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:46295 (local address 127.23.200.62:46603)
0806 20:38:54.127569 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:54.127577 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:54.127850 (+ 273us) server_negotiation.cc:407] Connection header received
0806 20:38:54.128051 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:54.128059 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:54.128150 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:54.128257 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:54.129266 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.130160 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.131557 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.132070 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.132213 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:54.137205 (+ 4992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:46295: BlockingRecv error: recv got EOF from 127.23.200.3:46295 (error 108)
Metrics: {"server-negotiator.queue_time_us":114}
Aug 06 20:38:54 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:54 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:54.533623 1628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:54.524087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:59355)
0806 20:38:54.524617 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:54.524634 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:54.524748 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:54.525053 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:54.525056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:54.525459 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:54.525627 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:54.525636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.526628 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.526632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:54.527552 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:54.527562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.528334 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.528340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:54.528585 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:54.529363 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:54.529388 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:54.533396 (+ 4008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
W20260806 20:38:54.533830 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:54.524138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:59355 (local address 127.23.200.62:46603)
0806 20:38:54.524339 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:54.524343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:54.524704 (+ 361us) server_negotiation.cc:407] Connection header received
0806 20:38:54.524843 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:54.524846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:54.524913 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:54.524994 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:54.525749 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.526501 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.527666 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.528168 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.528303 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:54.533579 (+ 5276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:59355: BlockingRecv error: recv got EOF from 127.23.200.1:59355 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 20:38:54 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:54 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:54.947887 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:54.938818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:41351)
0806 20:38:54.939371 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:54.939387 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:54.939512 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:54.939825 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:54.939828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:54.940210 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:54.940371 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:54.940379 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.941494 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.941498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:54.942416 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:54.942427 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.943173 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.943177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:54.943293 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:54.943962 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:54.943979 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:54.947660 (+ 3681us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":297,"threads_started":1}
W20260806 20:38:54.948153 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:54.938903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:41351 (local address 127.23.200.62:46603)
0806 20:38:54.939082 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:54.939086 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:54.939492 (+ 406us) server_negotiation.cc:407] Connection header received
0806 20:38:54.939617 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:54.939621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:54.939686 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:54.939779 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:54.940537 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.941361 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.942564 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:54.943036 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:54.943143 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:54.947895 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:41351: BlockingRecv error: recv got EOF from 127.23.200.2:41351 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 20:38:55 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:55 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:55.147243 1630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:55.138541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:33731)
0806 20:38:55.139062 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:55.139079 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:55.139236 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:55.139611 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:55.139618 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:55.140085 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:55.140250 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:55.140259 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.141321 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.141327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:55.142256 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:55.142267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.142998 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.143002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:55.143129 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:55.143705 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:55.143722 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:55.147008 (+ 3286us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":288,"threads_started":1}
W20260806 20:38:55.147445 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:55.138661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:33731 (local address 127.23.200.62:46603)
0806 20:38:55.138937 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:55.138944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:55.139168 (+ 224us) server_negotiation.cc:407] Connection header received
0806 20:38:55.139370 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:55.139382 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:55.139458 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:55.139568 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:55.140406 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.141172 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.142453 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.142864 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.142962 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:55.147213 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:33731: BlockingRecv error: recv got EOF from 127.23.200.3:33731 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 20:38:55 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:55 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:55.543540 1631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:55.534603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:54387)
0806 20:38:55.535249 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:55.535266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:55.535379 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:55.535663 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:55.535666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:55.536133 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:55.536333 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:55.536342 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.537381 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.537384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:55.538535 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:55.538547 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.539304 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.539323 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:55.539435 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:55.540014 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:55.540030 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:55.543239 (+ 3209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":300,"threads_started":1}
W20260806 20:38:55.543658 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:55.534684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:54387 (local address 127.23.200.62:46603)
0806 20:38:55.534866 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:55.534871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:55.535345 (+ 474us) server_negotiation.cc:407] Connection header received
0806 20:38:55.535469 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:55.535475 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:55.535538 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:55.535622 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:55.536477 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.537258 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.538651 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.539134 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.539259 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:55.543427 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:54387: BlockingRecv error: recv got EOF from 127.23.200.1:54387 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 20:38:55 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:55 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:55.958101 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:55.948725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:42029)
0806 20:38:55.949362 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:55.949378 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:55.949540 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:55.949880 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:55.949885 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:55.950404 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:55.950599 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:55.950608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.951656 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.951664 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:55.952621 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:55.952633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.953393 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.953399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:55.953584 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:55.954210 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:55.954228 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:55.957725 (+ 3497us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":267,"threads_started":1}
W20260806 20:38:55.958199 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:55.948812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:42029 (local address 127.23.200.62:46603)
0806 20:38:55.949020 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:55.949029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:55.949465 (+ 436us) server_negotiation.cc:407] Connection header received
0806 20:38:55.949670 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:55.949681 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:55.949744 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:55.949844 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:55.950749 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.951512 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.952767 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:55.953226 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:55.953362 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:55.957940 (+ 4578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:42029: BlockingRecv error: recv got EOF from 127.23.200.2:42029 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 06 20:38:56 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:56 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:56.157490 1633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:56.148193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:59597)
0806 20:38:56.148814 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:56.148830 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:56.148992 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:56.149399 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:56.149404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:56.149870 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:56.150044 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:56.150054 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.151169 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.151174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:56.152147 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:56.152160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.152895 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.152901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:56.153071 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:56.153723 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:56.153741 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:56.157125 (+ 3384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":289,"threads_started":1}
W20260806 20:38:56.157585 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:56.148331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:59597 (local address 127.23.200.62:46603)
0806 20:38:56.148553 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:56.148560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:56.148926 (+ 366us) server_negotiation.cc:407] Connection header received
0806 20:38:56.149124 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:56.149136 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:56.149254 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:56.149359 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:56.150196 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.151000 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.152299 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.152738 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.152862 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:56.157318 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:59597: BlockingRecv error: recv got EOF from 127.23.200.3:59597 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 20:38:56 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:56 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:56.557320 1634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:56.544678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:59707)
0806 20:38:56.545246 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:56.545261 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:56.545387 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:56.545680 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:56.545683 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:56.546121 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:56.546292 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:56.546328 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.547301 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.547304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:56.548226 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:56.548236 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.548940 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.548944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:56.549064 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:56.553312 (+ 4248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:56.553334 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:56.557072 (+ 3738us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":299,"threads_started":1}
W20260806 20:38:56.557514 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:56.544867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:59707 (local address 127.23.200.62:46603)
0806 20:38:56.545089 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:56.545096 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:56.545352 (+ 256us) server_negotiation.cc:407] Connection header received
0806 20:38:56.545487 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:56.545491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:56.545559 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:56.545632 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:56.546438 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.547189 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.548349 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.548786 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.548879 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:56.557251 (+ 8372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:59707: BlockingRecv error: recv got EOF from 127.23.200.1:59707 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Aug 06 20:38:56 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:56 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:56.968923 1635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:56.959312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:35933)
0806 20:38:56.959926 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:56.959943 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:56.960122 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:56.960511 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:56.960517 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:56.960975 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:56.961159 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:56.961168 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.962434 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.962440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:56.963392 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:56.963404 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.964153 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.964159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:56.964337 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:56.964996 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:56.965014 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:56.968569 (+ 3555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":357,"threads_started":1}
W20260806 20:38:56.969075 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:56.959391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:35933 (local address 127.23.200.62:46603)
0806 20:38:56.959727 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:56.959736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:56.960037 (+ 301us) server_negotiation.cc:407] Connection header received
0806 20:38:56.960256 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:56.960262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:56.960359 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:56.960475 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:56.961312 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.962237 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.963550 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:56.963956 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:56.964096 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:56.968796 (+ 4700us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:35933: BlockingRecv error: recv got EOF from 127.23.200.2:35933 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
Aug 06 20:38:57 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:57 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:57.167341 1636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:57.158686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:50763)
0806 20:38:57.159201 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:57.159218 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:57.159349 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:57.159656 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:57.159659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:57.160077 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:57.160243 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:57.160251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.161240 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.161244 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:57.162198 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:57.162210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.162938 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.162943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:57.163059 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:57.163640 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:57.163659 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:57.167098 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":298,"threads_started":1}
W20260806 20:38:57.167558 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:57.158782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:50763 (local address 127.23.200.62:46603)
0806 20:38:57.159031 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:57.159037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:57.159297 (+ 260us) server_negotiation.cc:407] Connection header received
0806 20:38:57.159446 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:57.159450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:57.159532 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:57.159610 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:57.160369 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.161086 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.162367 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.162776 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.162869 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:57.167285 (+ 4416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:50763: BlockingRecv error: recv got EOF from 127.23.200.3:50763 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 06 20:38:57 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:57 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:57.568593 1637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:57.558494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:38399)
0806 20:38:57.559073 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:57.559090 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:57.559210 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:57.559539 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:57.559542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:57.559973 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:57.560135 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:57.560144 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.561529 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.561534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:57.562736 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:57.562751 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.563467 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.563471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:57.563603 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:57.564339 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:57.564359 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:57.568222 (+ 3863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":315,"threads_started":1}
W20260806 20:38:57.568698 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:57.558566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38399 (local address 127.23.200.62:46603)
0806 20:38:57.558777 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:57.558781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:57.559220 (+ 439us) server_negotiation.cc:407] Connection header received
0806 20:38:57.559306 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:57.559310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:57.559384 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:57.559473 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:57.560311 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.561352 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.562879 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.563307 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.563408 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:57.568442 (+ 5034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:38399: BlockingRecv error: recv got EOF from 127.23.200.1:38399 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Aug 06 20:38:57 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:57 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:57.978818 1639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:57.969963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:44161)
0806 20:38:57.970514 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:57.970530 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:57.970680 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:57.971048 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:57.971052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:57.971482 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:57.971630 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:57.971639 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.972677 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.972684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:57.973637 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:57.973648 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.974411 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.974417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:57.974599 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:57.975134 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:57.975151 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:57.978587 (+ 3436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":266,"threads_started":1}
W20260806 20:38:57.979007 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:57.970012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:44161 (local address 127.23.200.62:46603)
0806 20:38:57.970244 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:57.970252 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:57.970613 (+ 361us) server_negotiation.cc:407] Connection header received
0806 20:38:57.970810 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:57.970815 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:57.970885 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:57.970990 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:57.971769 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.972538 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.973778 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:57.974236 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:57.974394 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:57.978761 (+ 4367us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:44161: BlockingRecv error: recv got EOF from 127.23.200.2:44161 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 20:38:58 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:58 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:58.178777 1640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:58.168518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:43551)
0806 20:38:58.169045 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:58.169063 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:58.169188 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:58.169565 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:58.169569 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:58.169997 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:58.170149 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:58.170158 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.171284 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.171289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:58.172397 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:58.172411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.173242 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.173247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:58.173416 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:58.174168 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:58.174188 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:58.178431 (+ 4243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":295,"threads_started":1}
W20260806 20:38:58.178885 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:58.168564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:43551 (local address 127.23.200.62:46603)
0806 20:38:58.168850 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:58.168857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:58.169156 (+ 299us) server_negotiation.cc:407] Connection header received
0806 20:38:58.169298 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:58.169303 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:58.169394 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:58.169487 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:58.170289 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.171112 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.172562 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.173084 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.173192 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:58.178653 (+ 5461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:43551: BlockingRecv error: recv got EOF from 127.23.200.3:43551 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 20:38:58 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:58 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:58.578850 1641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:58.569675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:33515)
0806 20:38:58.570207 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:58.570223 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:58.570465 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:58.570839 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:58.570843 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:58.571225 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:58.571399 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:58.571408 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.572402 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.572406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:58.573405 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:58.573416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.574291 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.574295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:58.574504 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:58.575213 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:58.575230 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:58.578621 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":288,"threads_started":1}
W20260806 20:38:58.579090 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:58.569744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:33515 (local address 127.23.200.62:46603)
0806 20:38:58.570025 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:58.570032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:58.570339 (+ 307us) server_negotiation.cc:407] Connection header received
0806 20:38:58.570628 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:58.570633 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:58.570716 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:58.570794 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:58.571534 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.572259 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.573557 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.574141 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.574276 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:58.578850 (+ 4574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:33515: BlockingRecv error: recv got EOF from 127.23.200.1:33515 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 06 20:38:58 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:58 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:58.989818 1642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:58.979911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:60905)
0806 20:38:58.980465 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:58.980481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:58.980662 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:58.981031 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:58.981036 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:58.981452 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:58.981616 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:58.981625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.982782 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.982789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:58.983786 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:58.983798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.984502 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.984507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:58.984682 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:58.985312 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:58.985330 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:58.989469 (+ 4139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":308,"threads_started":1}
W20260806 20:38:58.989969 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:58.979990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:60905 (local address 127.23.200.62:46603)
0806 20:38:58.980248 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:58.980255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:58.980582 (+ 327us) server_negotiation.cc:407] Connection header received
0806 20:38:58.980801 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:58.980806 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:58.980877 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:58.980984 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:58.981776 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.982647 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.983929 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:58.984341 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:58.984493 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:58.989708 (+ 5215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:60905: BlockingRecv error: recv got EOF from 127.23.200.2:60905 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 06 20:38:59 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:59 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:59.189683 1643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:59.180004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:53639)
0806 20:38:59.180594 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:59.180634 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:59.180785 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:59.181237 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:59.181242 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:59.181774 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:59.181970 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:59.181980 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.183145 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.183149 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:59.184240 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:59.184253 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.185217 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.185222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:59.185340 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:59.185930 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:59.185947 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:59.189430 (+ 3483us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":292,"threads_started":1}
W20260806 20:38:59.189882 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:59.180326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:53639 (local address 127.23.200.62:46603)
0806 20:38:59.180553 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:59.180559 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:59.180738 (+ 179us) server_negotiation.cc:407] Connection header received
0806 20:38:59.180945 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:59.180950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:59.181065 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:59.181173 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:59.182120 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.182983 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.184387 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.185070 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.185225 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:59.189616 (+ 4391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:53639: BlockingRecv error: recv got EOF from 127.23.200.3:53639 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Aug 06 20:38:59 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:59 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:38:59.588812 1644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:59.579919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:36751)
0806 20:38:59.580424 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:59.580441 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:59.580563 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:59.580845 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:59.580848 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:59.581269 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:59.581487 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:59.581495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.582478 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.582483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:59.583415 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:59.583426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.584071 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.584074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:59.584196 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:59.584774 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:59.584791 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:38:59.588484 (+ 3693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":273,"threads_started":1}
W20260806 20:38:59.588922 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:59.580109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:36751 (local address 127.23.200.62:46603)
0806 20:38:59.580340 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:59.580349 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:59.580530 (+ 181us) server_negotiation.cc:407] Connection header received
0806 20:38:59.580661 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:59.580665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:59.580725 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:59.580798 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:59.581612 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.582358 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.583539 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.583947 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.584034 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:59.588696 (+ 4662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:36751: BlockingRecv error: recv got EOF from 127.23.200.1:36751 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Aug 06 20:38:59 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:38:59 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:00.000546 1645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:38:59.990927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:56881)
0806 20:38:59.991899 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 20:38:59.991916 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:38:59.992094 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:38:59.992426 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:38:59.992431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:38:59.992901 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:38:59.993109 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:59.993117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.994217 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.994225 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:59.995282 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:38:59.995294 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.995995 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.996009 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:38:59.996192 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:38:59.996821 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:38:59.996838 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:00.000218 (+ 3380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":846,"thread_start_us":732,"threads_started":1}
W20260806 20:39:00.000669 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:38:59.991026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:56881 (local address 127.23.200.62:46603)
0806 20:38:59.991240 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 20:38:59.991246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:38:59.992016 (+ 770us) server_negotiation.cc:407] Connection header received
0806 20:38:59.992219 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:38:59.992225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:38:59.992304 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:38:59.992400 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:38:59.993244 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.994098 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.995416 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:38:59.995841 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:38:59.995978 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:00.000414 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:56881: BlockingRecv error: recv got EOF from 127.23.200.2:56881 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 06 20:39:00 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:00 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:00.200367 1646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:00.190760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:51469)
0806 20:39:00.191341 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:00.191357 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:00.191506 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:00.191806 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:00.191810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:00.192231 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:00.192414 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:00.192422 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.193648 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.193654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:00.194944 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:00.194962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.195833 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.195841 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:00.196018 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:00.196590 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:00.196607 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:00.200156 (+ 3549us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":329,"threads_started":1}
W20260806 20:39:00.200650 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:00.190825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:51469 (local address 127.23.200.62:46603)
0806 20:39:00.190989 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:00.190993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:00.191423 (+ 430us) server_negotiation.cc:407] Connection header received
0806 20:39:00.191619 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:00.191623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:00.191685 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:00.191766 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:00.192531 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.193468 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.195081 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.195625 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.195761 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:00.200347 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:51469: BlockingRecv error: recv got EOF from 127.23.200.3:51469 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 06 20:39:00 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:00 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:00.599202 1647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:00.589753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:38601)
0806 20:39:00.590297 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:00.590374 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:00.590512 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:00.590814 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:00.590816 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:00.591202 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:00.591368 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:00.591420 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.592440 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.592444 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:00.593460 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:00.593474 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.594252 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.594256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:00.594477 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:00.595123 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:00.595141 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:00.598867 (+ 3726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":301,"threads_started":1}
W20260806 20:39:00.599436 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:00.589828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38601 (local address 127.23.200.62:46603)
0806 20:39:00.590017 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:00.590021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:00.590445 (+ 424us) server_negotiation.cc:407] Connection header received
0806 20:39:00.590628 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:00.590632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:00.590694 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:00.590776 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:00.591562 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.592301 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.593586 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:00.594106 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:00.594211 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:00.599085 (+ 4874us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:38601: BlockingRecv error: recv got EOF from 127.23.200.1:38601 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 20:39:01 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:01 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:01.010385 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:01.001420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:42117)
0806 20:39:01.001951 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:01.001968 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:01.002084 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:01.002424 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:01.002428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:01.002811 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:01.003027 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:01.003040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.004447 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.004451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:01.005375 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:01.005387 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.005983 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.005987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:01.006115 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:01.006792 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:01.006810 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:01.010078 (+ 3268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":288,"threads_started":1}
W20260806 20:39:01.010551 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:01.001524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:42117 (local address 127.23.200.62:46603)
0806 20:39:01.001695 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:01.001699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:01.002043 (+ 344us) server_negotiation.cc:407] Connection header received
0806 20:39:01.002186 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:01.002190 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:01.002259 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:01.002362 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:01.003202 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.004308 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.005494 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.005865 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.005954 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:01.010258 (+ 4304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:42117: BlockingRecv error: recv got EOF from 127.23.200.2:42117 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Aug 06 20:39:01 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:01 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:01.210816 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:01.201388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:45903)
0806 20:39:01.202017 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:01.202043 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:01.202170 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:01.202622 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:01.202627 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:01.203217 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:01.203547 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:01.203560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.204667 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.204671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:01.205634 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:01.205645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.206406 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.206416 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:01.206549 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:01.207134 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:01.207152 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:01.210571 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":375,"threads_started":1}
W20260806 20:39:01.210996 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:01.201533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:45903 (local address 127.23.200.62:46603)
0806 20:39:01.201738 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:01.201745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:01.202127 (+ 382us) server_negotiation.cc:407] Connection header received
0806 20:39:01.202359 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:01.202370 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:01.202457 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:01.202557 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:01.203767 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.204533 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.205751 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.206165 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.206253 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:01.210757 (+ 4504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:45903: BlockingRecv error: recv got EOF from 127.23.200.3:45903 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 20:39:01 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:01 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:01.615149 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:01.600536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:45239)
0806 20:39:01.600987 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:01.601003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:01.601120 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:01.601401 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:01.601404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:01.601790 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:01.601941 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:01.601949 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.603001 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.603005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:01.603962 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:01.603974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.604705 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.604709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:01.604843 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:01.611529 (+ 6686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:01.611549 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:01.614905 (+ 3356us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":230,"threads_started":1}
W20260806 20:39:01.615345 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:01.600540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:45239 (local address 127.23.200.62:46603)
0806 20:39:01.600758 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:01.600767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:01.601086 (+ 319us) server_negotiation.cc:407] Connection header received
0806 20:39:01.601206 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:01.601209 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:01.601270 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:01.601352 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:01.602067 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.602854 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.604125 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:01.604555 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:01.604657 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:01.615119 (+ 10462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:45239: BlockingRecv error: recv got EOF from 127.23.200.1:45239 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 20:39:02 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:02 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.2@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:02.020871 1651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:02.011356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.2:49723)
0806 20:39:02.011981 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:02.012008 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:02.012183 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:02.012599 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:02.012603 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:02.013030 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:02.013174 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:02.013182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.014560 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.014565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:02.015473 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:02.015484 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.016401 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.016407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:02.016603 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.017239 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:02.017257 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:02.020628 (+ 3371us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":380,"threads_started":1}
W20260806 20:39:02.021082 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:02.011566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:49723 (local address 127.23.200.62:46603)
0806 20:39:02.011770 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:02.011779 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:02.012115 (+ 336us) server_negotiation.cc:407] Connection header received
0806 20:39:02.012325 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:02.012331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:02.012435 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:02.012557 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:02.013328 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.014421 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.015642 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.016248 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.016392 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.020828 (+ 4436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.2:49723: BlockingRecv error: recv got EOF from 127.23.200.2:49723 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 20:39:02 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:02 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.3@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:02.220661 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:02.211890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.3:52151)
0806 20:39:02.212626 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:02.212643 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:02.212761 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:02.213063 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:02.213066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:02.213503 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:02.213649 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:02.213658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.214745 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.214748 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:02.215652 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:02.215663 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.216306 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.216309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:02.216454 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.217034 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:02.217052 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:02.220416 (+ 3364us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":515,"threads_started":1}
W20260806 20:39:02.220901 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:02.211968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:52151 (local address 127.23.200.62:46603)
0806 20:39:02.212228 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:02.212237 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:02.212736 (+ 499us) server_negotiation.cc:407] Connection header received
0806 20:39:02.212866 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:02.212869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:02.212933 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:02.213019 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:02.213786 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.214621 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.215781 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.216180 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.216273 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.220657 (+ 4384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.3:52151: BlockingRecv error: recv got EOF from 127.23.200.3:52151 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 06 20:39:02 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
Aug 06 20:39:02 dist-test-slave-tm1g krb5kdc[556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.200.1@KRBTEST.COM for oryx/127.23.200.62@KRBTEST.COM, Server not found in Kerberos database
I20260806 20:39:02.625494 1653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:02.616335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:46603 (local address 127.23.200.1:46773)
0806 20:39:02.616851 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:02.616867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:02.617034 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:02.617406 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:02.617409 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:02.617801 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:02.617979 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:02.617987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.619131 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.619135 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:02.620058 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:02.620070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.620782 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.620786 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:02.620913 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.621505 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:02.621523 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:02.625209 (+ 3686us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.200.62:46603: Server oryx/127.23.200.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":268,"threads_started":1}
W20260806 20:39:02.625674 1608 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:02.616398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:46773 (local address 127.23.200.62:46603)
0806 20:39:02.616735 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:02.616742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:02.616970 (+ 228us) server_negotiation.cc:407] Connection header received
0806 20:39:02.617169 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:02.617174 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:02.617256 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:02.617374 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:02.618134 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.619002 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.620182 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.620648 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.620745 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.625444 (+ 4699us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.200.1:46773: BlockingRecv error: recv got EOF from 127.23.200.1:46773 (error 108)
Metrics: {"server-negotiator.queue_time_us":189}
I20260806 20:39:02.930438 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1170
I20260806 20:39:02.943578 1655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:02.931853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43356 (local address 127.23.200.1:36507)
0806 20:39:02.932562 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:02.932569 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:02.932623 (+ 54us) server_negotiation.cc:407] Connection header received
0806 20:39:02.932687 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:02.932695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:02.932939 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:02.933113 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:02.934659 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.935740 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.936914 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.937445 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.937553 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.940392 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:02.940402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:02.940416 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:02.940474 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:02.943044 (+ 2570us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.200.1@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 20:39:02.943340 (+ 296us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43356: Request ticket server kudu/127.23.200.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":499,"thread_start_us":372,"threads_started":1}
W20260806 20:39:02.943710 1655 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43356: Request ticket server kudu/127.23.200.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 20:39:02.943732 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1170: Not authorized: Client connection negotiation failed: client connection to 127.23.200.1:36507: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.200.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 20:39:02.943984 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1170
I20260806 20:39:02.963368 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1330
I20260806 20:39:02.976425 1656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:02.964574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.23.200.2:46485)
0806 20:39:02.965186 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:02.965194 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:02.965221 (+ 27us) server_negotiation.cc:407] Connection header received
0806 20:39:02.965288 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:02.965308 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:02.965593 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:02.965830 (+ 237us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:02.967279 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.968316 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.969584 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.970148 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.970307 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:02.972886 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:02.972896 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:02.972910 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:02.972951 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:02.975671 (+ 2720us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.200.2@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 20:39:02.976078 (+ 407us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60746: Request ticket server kudu/127.23.200.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":296,"threads_started":1}
W20260806 20:39:02.976449 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1330: Not authorized: Client connection negotiation failed: client connection to 127.23.200.2:46485: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.200.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 20:39:02.976565 1656 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60746: Request ticket server kudu/127.23.200.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 20:39:02.976634 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1330
I20260806 20:39:02.991940 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1469
I20260806 20:39:03.004900 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:02.993018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.23.200.3:33383)
0806 20:39:02.993647 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:02.993657 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:02.993687 (+ 30us) server_negotiation.cc:407] Connection header received
0806 20:39:02.993761 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:02.993771 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:02.994127 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:02.994370 (+ 243us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:02.995730 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.996766 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.998087 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:02.998767 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:02.998868 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:03.001393 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:03.001401 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:03.001415 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:03.001453 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:03.004324 (+ 2871us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.200.3@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 20:39:03.004608 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53742: Request ticket server kudu/127.23.200.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":283,"threads_started":1}
W20260806 20:39:03.005023 1657 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53742: Request ticket server kudu/127.23.200.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 20:39:03.005108 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1469: Not authorized: Client connection negotiation failed: client connection to 127.23.200.3:33383: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.23.200.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 20:39:03.005318 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1469
I20260806 20:39:03.023418 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1098
I20260806 20:39:03.024808 1163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:03.143249 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1098
2026-08-06T20:39:03Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21741 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T20:39:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:39:03Z Disabled control of system clock
I20260806 20:39:03.223141 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45497
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:45497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 20:39:03.547986 1662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:03.548260 1662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:03.548342 1662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:03.555256 1662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:39:03.555356 1662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:03.555393 1662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:03.555442 1662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:39:03.555487 1662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:39:03.564155 1662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:45497
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:03.565385 1662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:03.567108 1662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:03.578536 1669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:03.578591 1668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:03.578751 1671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:03.579084 1662 server_base.cc:1034] running on GCE node
I20260806 20:39:03.579972 1662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:03.581341 1662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:03.582571 1662 hybrid_clock.cc:648] HybridClock initialized: now 1786048743582540 us; error 42 us; skew 500 ppm
I20260806 20:39:03.582917 1662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:03.585016 1662 webserver.cc:460] Webserver started at http://127.23.200.62:46815/ using document root <none> and password file <none>
I20260806 20:39:03.585714 1662 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:03.585846 1662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:03.586175 1662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:03.589020 1662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "12eedba505e344d697857f3a71651351"
format_stamp: "Formatted at 2026-08-06 20:39:03 on dist-test-slave-tm1g"
server_key: "0a23469790a147cb8874e7e76a24c296"
server_key_iv: "dffc09fd97e58b9f0e25cc544b5acd4a"
server_key_version: "encryptionkey@0"
I20260806 20:39:03.589792 1662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "12eedba505e344d697857f3a71651351"
format_stamp: "Formatted at 2026-08-06 20:39:03 on dist-test-slave-tm1g"
server_key: "0a23469790a147cb8874e7e76a24c296"
server_key_iv: "dffc09fd97e58b9f0e25cc544b5acd4a"
server_key_version: "encryptionkey@0"
I20260806 20:39:03.594837 1662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:39:03.598158 1677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:03.599538 1662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:39:03.599738 1662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "12eedba505e344d697857f3a71651351"
format_stamp: "Formatted at 2026-08-06 20:39:03 on dist-test-slave-tm1g"
server_key: "0a23469790a147cb8874e7e76a24c296"
server_key_iv: "dffc09fd97e58b9f0e25cc544b5acd4a"
server_key_version: "encryptionkey@0"
I20260806 20:39:03.599964 1662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:03.615518 1662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:03.616317 1662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:03.616612 1662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:03.634898 1662 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:45497
I20260806 20:39:03.634907 1731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:45497 every 8 connection(s)
I20260806 20:39:03.636256 1662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:39:03.639923 1732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:03.643678 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1662
I20260806 20:39:03.643908 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:39:03.644733 24352 external_mini_cluster.cc:1428] Setting key 20096cbdba8b6de1a25ecdcd400ee8bc
I20260806 20:39:03.653501 1732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351: Bootstrap starting.
I20260806 20:39:03.657235 1732 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351: Neither blocks nor log segments found. Creating new log.
I20260806 20:39:03.658421 1732 log.cc:824] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:03.658946 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:03.646972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52828 (local address 127.23.200.62:45497)
0806 20:39:03.647953 (+ 981us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:03.647967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:03.648012 (+ 45us) server_negotiation.cc:407] Connection header received
0806 20:39:03.649575 (+ 1563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:03.649597 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:03.650086 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:03.650512 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:03.651094 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:03.652337 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:03.654192 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:03.654917 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:03.655065 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:03.656391 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:03.656398 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:03.656411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:03.656958 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:03.657048 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:03.657074 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:03.657163 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:03.657500 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 20:39:03.657844 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":374,"threads_started":1}
I20260806 20:39:03.661836 1732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351: No bootstrap required, opened a new log
I20260806 20:39:03.669916 1732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eedba505e344d697857f3a71651351" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45497 } }
I20260806 20:39:03.670480 1732 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:03.670614 1732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12eedba505e344d697857f3a71651351, State: Initialized, Role: FOLLOWER
I20260806 20:39:03.671293 1732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eedba505e344d697857f3a71651351" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45497 } }
I20260806 20:39:03.671603 1732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:39:03.671762 1732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:39:03.671963 1732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:03.674446 1732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eedba505e344d697857f3a71651351" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45497 } }
I20260806 20:39:03.674952 1732 leader_election.cc:302] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 12eedba505e344d697857f3a71651351; no voters:
I20260806 20:39:03.675586 1732 leader_election.cc:288] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:39:03.675750 1737 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:39:03.676482 1737 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [term 1 LEADER]: Becoming Leader. State: Replica: 12eedba505e344d697857f3a71651351, State: Running, Role: LEADER
I20260806 20:39:03.677018 1737 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eedba505e344d697857f3a71651351" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45497 } }
I20260806 20:39:03.677735 1732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:39:03.679791 1739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12eedba505e344d697857f3a71651351. Latest consensus state: current_term: 1 leader_uuid: "12eedba505e344d697857f3a71651351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eedba505e344d697857f3a71651351" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45497 } } }
I20260806 20:39:03.680138 1739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:03.680912 1738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "12eedba505e344d697857f3a71651351" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "12eedba505e344d697857f3a71651351" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45497 } } }
I20260806 20:39:03.681334 1738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351 [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:03.683175 1743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:39:03.687996 1743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:39:03.696097 1743 catalog_manager.cc:1347] Generated new cluster ID: a724fdd98c0c46eea780bf68d819a091
I20260806 20:39:03.696213 1743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:39:03.737602 1743 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:39:03.739015 1743 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:39:03.752898 1743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 12eedba505e344d697857f3a71651351: Generated new TSK 0
I20260806 20:39:03.753818 1743 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:39:03.761171 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45497
--builtin_ntp_servers=127.23.200.20:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 20:39:04.099000 1758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:04.099280 1758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:04.099445 1758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:04.107030 1758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:04.107123 1758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:04.107319 1758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:39:04.116227 1758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:04.117578 1758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:04.119452 1758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:04.132743 1764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:04.132800 1763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:04.132826 1766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:04.133344 1758 server_base.cc:1034] running on GCE node
I20260806 20:39:04.134032 1758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:04.134997 1758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:04.136193 1758 hybrid_clock.cc:648] HybridClock initialized: now 1786048744136146 us; error 35 us; skew 500 ppm
I20260806 20:39:04.136643 1758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:04.138800 1758 webserver.cc:460] Webserver started at http://127.23.200.1:39403/ using document root <none> and password file <none>
I20260806 20:39:04.139448 1758 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:04.139534 1758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:04.139808 1758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:04.142567 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "e7ca73ad7b0d43c4b9d948f67509ab63"
format_stamp: "Formatted at 2026-08-06 20:39:04 on dist-test-slave-tm1g"
server_key: "9a87a775419d2931fa70401ed8745346"
server_key_iv: "9300d958e2edd753c7cca3fcc85e1830"
server_key_version: "encryptionkey@0"
I20260806 20:39:04.143318 1758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "e7ca73ad7b0d43c4b9d948f67509ab63"
format_stamp: "Formatted at 2026-08-06 20:39:04 on dist-test-slave-tm1g"
server_key: "9a87a775419d2931fa70401ed8745346"
server_key_iv: "9300d958e2edd753c7cca3fcc85e1830"
server_key_version: "encryptionkey@0"
I20260806 20:39:04.148202 1758 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:39:04.151455 1772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:04.152798 1758 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:39:04.152978 1758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "e7ca73ad7b0d43c4b9d948f67509ab63"
format_stamp: "Formatted at 2026-08-06 20:39:04 on dist-test-slave-tm1g"
server_key: "9a87a775419d2931fa70401ed8745346"
server_key_iv: "9300d958e2edd753c7cca3fcc85e1830"
server_key_version: "encryptionkey@0"
I20260806 20:39:04.153206 1758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:04.190039 1758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:04.191097 1758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:04.191432 1758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:04.192476 1758 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:04.193991 1758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:04.194124 1758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:04.194258 1758 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:04.194370 1758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 20:39:04.228183 1758 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:46059
I20260806 20:39:04.228276 1887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:46059 every 8 connection(s)
I20260806 20:39:04.229604 1758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:39:04.238143 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1758
I20260806 20:39:04.238400 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:39:04.239040 24352 external_mini_cluster.cc:1428] Setting key b0ad8d5f6bb7031bd05a6a34f25e796c
I20260806 20:39:04.242064 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45497
--builtin_ntp_servers=127.23.200.20:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 20:39:04.243156 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:04.232301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49049 (local address 127.23.200.62:45497)
0806 20:39:04.232788 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:04.232793 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:04.233922 (+ 1129us) server_negotiation.cc:407] Connection header received
0806 20:39:04.235834 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:04.235839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:04.235937 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:04.236068 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:04.237170 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.237934 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.239792 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.240334 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.240489 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:04.242154 (+ 1665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:04.242162 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:04.242169 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:04.242196 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:04.242259 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:04.242275 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:04.242411 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:04.242666 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 20:39:04.242772 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":267,"threads_started":1}
I20260806 20:39:04.244220 1891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:04.232808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45497 (local address 127.23.200.1:49049)
0806 20:39:04.233746 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:04.233782 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:04.235539 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:04.236320 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:04.236329 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:04.236416 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:04.236990 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:04.237006 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.238111 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.238115 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:04.239636 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:04.239650 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.240587 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.240598 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:04.240920 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:04.241718 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:39:04.241721 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:04.241765 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:39:04.241854 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:04.242428 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:04.242442 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:04.242450 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 20:39:04.242651 (+ 201us) client_negotiation.cc:240] Negotiation successful
0806 20:39:04.243086 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":274,"threads_started":1}
I20260806 20:39:04.246011 1888 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45497
I20260806 20:39:04.246421 1888 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:04.247606 1888 heartbeater.cc:511] Master 127.23.200.62:45497 requested a full tablet report, sending...
I20260806 20:39:04.258157 1694 ts_manager.cc:194] Registered new tserver with Master: e7ca73ad7b0d43c4b9d948f67509ab63 (127.23.200.1:46059)
I20260806 20:39:04.260509 1694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.1:49049
W20260806 20:39:04.595359 1892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:04.595768 1892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:04.595938 1892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:04.603307 1892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:04.603462 1892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:04.603631 1892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:39:04.613085 1892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:04.614542 1892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:04.616389 1892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:04.629436 1897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:04.629554 1898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:04.629714 1900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:04.630699 1892 server_base.cc:1034] running on GCE node
I20260806 20:39:04.631330 1892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:04.632355 1892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:04.633612 1892 hybrid_clock.cc:648] HybridClock initialized: now 1786048744633563 us; error 61 us; skew 500 ppm
I20260806 20:39:04.633944 1892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:04.636816 1892 webserver.cc:460] Webserver started at http://127.23.200.2:43641/ using document root <none> and password file <none>
I20260806 20:39:04.637681 1892 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:04.637796 1892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:04.638168 1892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:04.641968 1892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "9c2b310ad89f44cda4697ca8dd5f60e4"
format_stamp: "Formatted at 2026-08-06 20:39:04 on dist-test-slave-tm1g"
server_key: "9a3504d27f6d2b885353908582df2aaa"
server_key_iv: "31b671e86924eb47a73461fbf233b0fd"
server_key_version: "encryptionkey@0"
I20260806 20:39:04.642964 1892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "9c2b310ad89f44cda4697ca8dd5f60e4"
format_stamp: "Formatted at 2026-08-06 20:39:04 on dist-test-slave-tm1g"
server_key: "9a3504d27f6d2b885353908582df2aaa"
server_key_iv: "31b671e86924eb47a73461fbf233b0fd"
server_key_version: "encryptionkey@0"
I20260806 20:39:04.649528 1892 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 20:39:04.654006 1906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:04.655709 1892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:39:04.655865 1892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "9c2b310ad89f44cda4697ca8dd5f60e4"
format_stamp: "Formatted at 2026-08-06 20:39:04 on dist-test-slave-tm1g"
server_key: "9a3504d27f6d2b885353908582df2aaa"
server_key_iv: "31b671e86924eb47a73461fbf233b0fd"
server_key_version: "encryptionkey@0"
I20260806 20:39:04.656095 1892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:04.684144 1892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:04.685088 1892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:04.685461 1892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:04.686571 1892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:04.688056 1892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:04.688189 1892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:04.688324 1892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:04.688426 1892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:04.722391 1892 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33469
I20260806 20:39:04.722505 2021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33469 every 8 connection(s)
I20260806 20:39:04.723742 1892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:39:04.730513 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 1892
I20260806 20:39:04.730702 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:39:04.731164 24352 external_mini_cluster.cc:1428] Setting key b01f2ef8554701a27979baafa8f50080
I20260806 20:39:04.734162 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45497
--builtin_ntp_servers=127.23.200.20:40371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 20:39:04.737006 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:04.726460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:60209 (local address 127.23.200.62:45497)
0806 20:39:04.726669 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:04.726673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:04.728067 (+ 1394us) server_negotiation.cc:407] Connection header received
0806 20:39:04.729881 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:04.729888 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:04.729981 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:04.730098 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:04.731372 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.732127 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.733842 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.734548 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.734690 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:04.736091 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:04.736099 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:04.736107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:04.736132 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:04.736175 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:04.736193 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:04.736283 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:04.736492 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 20:39:04.736603 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 20:39:04.737875 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:04.726978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45497 (local address 127.23.200.2:60209)
0806 20:39:04.727900 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:04.727941 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:04.729589 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:04.730395 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:04.730404 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:04.730500 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:04.731171 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:04.731188 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.732295 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.732300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:04.733669 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:04.733681 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:04.734796 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:04.734802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:04.735017 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:04.735751 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:39:04.735753 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:04.735797 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:39:04.735881 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:04.736288 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:04.736296 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:04.736302 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 20:39:04.736450 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 20:39:04.736745 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":269,"threads_started":1}
I20260806 20:39:04.739372 2022 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45497
I20260806 20:39:04.739837 2022 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:04.740794 2022 heartbeater.cc:511] Master 127.23.200.62:45497 requested a full tablet report, sending...
I20260806 20:39:04.742532 1694 ts_manager.cc:194] Registered new tserver with Master: 9c2b310ad89f44cda4697ca8dd5f60e4 (127.23.200.2:33469)
I20260806 20:39:04.743782 1694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.2:60209
W20260806 20:39:05.069265 2025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:05.069543 2025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:05.069629 2025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:05.076844 2025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:05.076947 2025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:05.077059 2025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:39:05.086333 2025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45497
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:05.087608 2025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:05.089373 2025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:05.102389 2033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:05.102479 2030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:05.102451 2031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:05.107740 2025 server_base.cc:1034] running on GCE node
I20260806 20:39:05.108352 2025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:05.109316 2025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:05.110523 2025 hybrid_clock.cc:648] HybridClock initialized: now 1786048745110470 us; error 35 us; skew 500 ppm
I20260806 20:39:05.110817 2025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:05.113055 2025 webserver.cc:460] Webserver started at http://127.23.200.3:44099/ using document root <none> and password file <none>
I20260806 20:39:05.113742 2025 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:05.113828 2025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:05.114100 2025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:05.116940 2025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "1b67f9194cfd47af81a2c78f5bad4533"
format_stamp: "Formatted at 2026-08-06 20:39:05 on dist-test-slave-tm1g"
server_key: "09d1b89d93e185e5c3903434574bddc2"
server_key_iv: "183dc7a1427b57a9b9701b98d0cbaceb"
server_key_version: "encryptionkey@0"
I20260806 20:39:05.117686 2025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "1b67f9194cfd47af81a2c78f5bad4533"
format_stamp: "Formatted at 2026-08-06 20:39:05 on dist-test-slave-tm1g"
server_key: "09d1b89d93e185e5c3903434574bddc2"
server_key_iv: "183dc7a1427b57a9b9701b98d0cbaceb"
server_key_version: "encryptionkey@0"
I20260806 20:39:05.122783 2025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:39:05.126032 2039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:05.127333 2025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:39:05.127519 2025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "1b67f9194cfd47af81a2c78f5bad4533"
format_stamp: "Formatted at 2026-08-06 20:39:05 on dist-test-slave-tm1g"
server_key: "09d1b89d93e185e5c3903434574bddc2"
server_key_iv: "183dc7a1427b57a9b9701b98d0cbaceb"
server_key_version: "encryptionkey@0"
I20260806 20:39:05.127729 2025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:05.163527 2025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:05.164542 2025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:05.164896 2025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:05.166035 2025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:05.167651 2025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:05.167781 2025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:05.167925 2025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:05.168010 2025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:05.201436 2025 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:40219
I20260806 20:39:05.201534 2156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:40219 every 8 connection(s)
I20260806 20:39:05.202836 2025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:39:05.210883 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2025
I20260806 20:39:05.211127 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:39:05.211712 24352 external_mini_cluster.cc:1428] Setting key 23fb92b7b9cbafcfe9ba1e1e7d61f7e8
I20260806 20:39:05.217012 1890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:05.205931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:55183 (local address 127.23.200.62:45497)
0806 20:39:05.206185 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:05.206191 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:05.207727 (+ 1536us) server_negotiation.cc:407] Connection header received
0806 20:39:05.209470 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:05.209474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:05.209557 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:05.209655 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:05.210869 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.212129 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.214127 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.214678 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.214814 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:05.216165 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:05.216171 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:05.216176 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:05.216203 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:05.216235 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:05.216248 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:05.216344 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:05.216566 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 20:39:05.216662 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260806 20:39:05.217983 2160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:05.206624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45497 (local address 127.23.200.3:55183)
0806 20:39:05.207557 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:05.207595 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:05.209209 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:05.209898 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:05.209906 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:05.209990 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:05.210681 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:05.210699 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.212298 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.212302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:05.213985 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:05.214002 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.214903 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.214910 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:05.215143 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:05.215855 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:39:05.215857 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:05.215896 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:39:05.215970 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:05.216350 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:05.216364 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:05.216374 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 20:39:05.216557 (+ 183us) client_negotiation.cc:240] Negotiation successful
0806 20:39:05.216954 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":292,"threads_started":1}
I20260806 20:39:05.219245 2157 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45497
I20260806 20:39:05.219569 2157 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:05.220347 2157 heartbeater.cc:511] Master 127.23.200.62:45497 requested a full tablet report, sending...
I20260806 20:39:05.221827 1694 ts_manager.cc:194] Registered new tserver with Master: 1b67f9194cfd47af81a2c78f5bad4533 (127.23.200.3:40219)
I20260806 20:39:05.222728 1694 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.3:55183
I20260806 20:39:05.227496 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 20:39:05.235885 1890 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:05.234021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.23.200.62:45497)
0806 20:39:05.234295 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:05.234324 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:05.234827 (+ 503us) server_negotiation.cc:407] Connection header received
0806 20:39:05.235005 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:05.235009 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:05.235086 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:05.235170 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:05.235644 (+ 474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52842: BlockingRecv error: recv got EOF from 127.0.0.1:52842 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:39:05.236927 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1758
I20260806 20:39:05.245469 2167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:05.237880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57638 (local address 127.23.200.1:46059)
0806 20:39:05.238479 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:05.238487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:05.238513 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:39:05.238577 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:05.238583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:05.238785 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:05.238996 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:05.239465 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.240596 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.241987 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.242684 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.242839 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:05.243971 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:05.243980 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:05.243993 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:05.244581 (+ 588us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:05.244630 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:05.244656 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:05.244788 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:05.244947 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 20:39:05.245107 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":241,"threads_started":1}
I20260806 20:39:05.246812 1883 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:05.334728 1888 heartbeater.cc:503] Master 127.23.200.62:45497 was elected leader, sending a full tablet report...
I20260806 20:39:05.336066 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1758
W20260806 20:39:05.353633 1681 connection.cc:535] server connection from 127.23.200.1:49049 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 20:39:05.353950 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1892
I20260806 20:39:05.362190 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:05.354989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.23.200.2:33469)
0806 20:39:05.355484 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:05.355491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:05.355517 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:39:05.355578 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:05.355584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:05.355817 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:05.355993 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:05.356420 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.357434 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.358784 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.359363 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.359512 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:05.360644 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:05.360653 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:05.360666 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:05.361321 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:05.361370 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:05.361400 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:05.361532 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:05.361706 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 20:39:05.361841 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":238,"threads_started":1}
I20260806 20:39:05.363559 2017 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:05.448951 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1892
I20260806 20:39:05.466910 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2025
I20260806 20:39:05.475802 2171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:05.467928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.23.200.3:40219)
0806 20:39:05.468588 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:05.468595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:05.468626 (+ 31us) server_negotiation.cc:407] Connection header received
0806 20:39:05.468689 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:05.468696 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:05.468919 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:05.469118 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:05.469595 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.470816 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.472103 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:05.472706 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:05.472852 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:05.474082 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:05.474091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:05.474105 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:05.474885 (+ 780us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:05.474936 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:05.474956 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:05.475102 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:05.475269 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 20:39:05.475422 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":335,"threads_started":1}
I20260806 20:39:05.477195 2152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:05.561225 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2025
I20260806 20:39:05.578528 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 1662
I20260806 20:39:05.579890 1726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:05.684809 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 1662
2026-08-06T20:39:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2537 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T20:39:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:39:05Z Disabled control of system clock
I20260806 20:39:05.765162 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44817
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:44817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 20:39:06.102823 2177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:06.103107 2177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:06.103191 2177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:06.110482 2177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:39:06.110584 2177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:06.110622 2177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:06.110679 2177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:39:06.110716 2177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:39:06.119753 2177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:44817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:06.120983 2177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:06.122813 2177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:06.134841 2182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:06.134759 2183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:06.134985 2185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:06.135756 2177 server_base.cc:1034] running on GCE node
I20260806 20:39:06.136524 2177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:06.137941 2177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:06.139186 2177 hybrid_clock.cc:648] HybridClock initialized: now 1786048746139146 us; error 63 us; skew 500 ppm
I20260806 20:39:06.139567 2177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:06.141938 2177 webserver.cc:460] Webserver started at http://127.23.200.62:39861/ using document root <none> and password file <none>
I20260806 20:39:06.142728 2177 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:06.142864 2177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:06.143200 2177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:06.146095 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "771029189ed04cca820cee645868997f"
format_stamp: "Formatted at 2026-08-06 20:39:06 on dist-test-slave-tm1g"
server_key: "6ac2cd539f5d0bae8e39955af0b04d7a"
server_key_iv: "f74f1b0f0a3b2c5997cc4501ed278bf3"
server_key_version: "encryptionkey@0"
I20260806 20:39:06.146960 2177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "771029189ed04cca820cee645868997f"
format_stamp: "Formatted at 2026-08-06 20:39:06 on dist-test-slave-tm1g"
server_key: "6ac2cd539f5d0bae8e39955af0b04d7a"
server_key_iv: "f74f1b0f0a3b2c5997cc4501ed278bf3"
server_key_version: "encryptionkey@0"
I20260806 20:39:06.152045 2177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:39:06.155469 2191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:06.156692 2177 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:39:06.156880 2177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "771029189ed04cca820cee645868997f"
format_stamp: "Formatted at 2026-08-06 20:39:06 on dist-test-slave-tm1g"
server_key: "6ac2cd539f5d0bae8e39955af0b04d7a"
server_key_iv: "f74f1b0f0a3b2c5997cc4501ed278bf3"
server_key_version: "encryptionkey@0"
I20260806 20:39:06.157094 2177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:06.162905 2177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:06.163657 2177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:06.163969 2177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:06.182677 2177 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:44817
I20260806 20:39:06.182710 2246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:44817 every 8 connection(s)
I20260806 20:39:06.184017 2177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:39:06.187631 2247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:06.189038 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2177
I20260806 20:39:06.189231 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:39:06.189940 24352 external_mini_cluster.cc:1428] Setting key 40e8e779b5772184a413bf70da9a6750
I20260806 20:39:06.198383 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:06.192026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45448 (local address 127.23.200.62:44817)
0806 20:39:06.193124 (+ 1098us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:06.193135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:06.193179 (+ 44us) server_negotiation.cc:407] Connection header received
0806 20:39:06.194840 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:06.194861 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:06.195242 (+ 381us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:06.195562 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:06.196712 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:06.196718 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:06.196731 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:06.196746 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:06.196809 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:06.196825 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:06.197011 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:06.197130 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 20:39:06.197449 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":430,"threads_started":1}
I20260806 20:39:06.199558 2247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f: Bootstrap starting.
I20260806 20:39:06.202669 2247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f: Neither blocks nor log segments found. Creating new log.
I20260806 20:39:06.203678 2247 log.cc:824] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:06.206178 2247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f: No bootstrap required, opened a new log
I20260806 20:39:06.213081 2247 raft_consensus.cc:348] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771029189ed04cca820cee645868997f" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44817 } }
I20260806 20:39:06.213433 2247 raft_consensus.cc:374] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:06.213537 2247 raft_consensus.cc:729] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 771029189ed04cca820cee645868997f, State: Initialized, Role: FOLLOWER
I20260806 20:39:06.214069 2247 consensus_queue.cc:260] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771029189ed04cca820cee645868997f" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44817 } }
I20260806 20:39:06.214339 2247 raft_consensus.cc:388] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:39:06.214488 2247 raft_consensus.cc:482] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:39:06.214659 2247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:06.216692 2247 raft_consensus.cc:504] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771029189ed04cca820cee645868997f" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44817 } }
I20260806 20:39:06.217175 2247 leader_election.cc:302] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 771029189ed04cca820cee645868997f; no voters:
I20260806 20:39:06.217854 2247 leader_election.cc:288] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:39:06.217991 2252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:39:06.218793 2252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [term 1 LEADER]: Becoming Leader. State: Replica: 771029189ed04cca820cee645868997f, State: Running, Role: LEADER
I20260806 20:39:06.219349 2252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771029189ed04cca820cee645868997f" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44817 } }
I20260806 20:39:06.219956 2247 sys_catalog.cc:564] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:39:06.222463 2254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 771029189ed04cca820cee645868997f. Latest consensus state: current_term: 1 leader_uuid: "771029189ed04cca820cee645868997f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771029189ed04cca820cee645868997f" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44817 } } }
I20260806 20:39:06.222443 2253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "771029189ed04cca820cee645868997f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "771029189ed04cca820cee645868997f" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44817 } } }
I20260806 20:39:06.222913 2254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:06.222991 2253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:06.223991 2262 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:39:06.230355 2262 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:39:06.239053 2262 catalog_manager.cc:1347] Generated new cluster ID: 14728c5920474095a82bb8de088b26ee
I20260806 20:39:06.239166 2262 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:39:06.255635 2262 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:39:06.256101 2262 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:39:06.266458 2262 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 771029189ed04cca820cee645868997f: Generated new TSK 0
I20260806 20:39:06.267580 2262 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:39:06.283514 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:44817
--builtin_ntp_servers=127.23.200.20:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 20:39:06.636803 2272 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:06.637142 2272 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:06.637317 2272 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:06.644631 2272 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:06.644779 2272 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:06.644974 2272 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:39:06.654042 2272 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:44817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:06.655499 2272 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:06.657469 2272 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:06.671164 2277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:06.671197 2280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:06.671166 2278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:06.672441 2272 server_base.cc:1034] running on GCE node
I20260806 20:39:06.673032 2272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:06.674031 2272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:06.675266 2272 hybrid_clock.cc:648] HybridClock initialized: now 1786048746675187 us; error 100 us; skew 500 ppm
I20260806 20:39:06.675606 2272 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:06.680995 2272 webserver.cc:460] Webserver started at http://127.23.200.1:42925/ using document root <none> and password file <none>
I20260806 20:39:06.681770 2272 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:06.681911 2272 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:06.682260 2272 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:06.686969 2272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "40b8181285da43efb7d837cabff05bf0"
format_stamp: "Formatted at 2026-08-06 20:39:06 on dist-test-slave-tm1g"
server_key: "3b62956a7414b5913c84e5572289a189"
server_key_iv: "ce5f29b0ee131326579332560a02f980"
server_key_version: "encryptionkey@0"
I20260806 20:39:06.687810 2272 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "40b8181285da43efb7d837cabff05bf0"
format_stamp: "Formatted at 2026-08-06 20:39:06 on dist-test-slave-tm1g"
server_key: "3b62956a7414b5913c84e5572289a189"
server_key_iv: "ce5f29b0ee131326579332560a02f980"
server_key_version: "encryptionkey@0"
I20260806 20:39:06.693588 2272 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:39:06.697834 2286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:06.699250 2272 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 20:39:06.699425 2272 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "40b8181285da43efb7d837cabff05bf0"
format_stamp: "Formatted at 2026-08-06 20:39:06 on dist-test-slave-tm1g"
server_key: "3b62956a7414b5913c84e5572289a189"
server_key_iv: "ce5f29b0ee131326579332560a02f980"
server_key_version: "encryptionkey@0"
I20260806 20:39:06.699685 2272 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:06.705493 2272 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:06.706280 2272 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:06.706643 2272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:06.707556 2272 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:06.708925 2272 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:06.709056 2272 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:06.709177 2272 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:06.709260 2272 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:06.743777 2272 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:45055
I20260806 20:39:06.743865 2401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:45055 every 8 connection(s)
I20260806 20:39:06.745143 2272 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:39:06.751999 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2272
I20260806 20:39:06.752194 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:39:06.752934 24352 external_mini_cluster.cc:1428] Setting key 1148bf405e3e9fbb16aecf7d08a38ba3
I20260806 20:39:06.753657 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:06.747798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:47117 (local address 127.23.200.62:44817)
0806 20:39:06.748268 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:06.748274 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:06.749500 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 20:39:06.751259 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:06.751264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:06.751330 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:06.751434 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:06.752787 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:06.752795 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:06.752800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:06.752821 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:06.752879 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:06.752897 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:06.753051 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:06.753275 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 20:39:06.753356 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":223,"threads_started":1}
I20260806 20:39:06.754647 2405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:06.748424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44817 (local address 127.23.200.1:47117)
0806 20:39:06.749339 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:06.749387 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:06.750993 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:06.751645 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:06.751653 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:06.751742 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:06.752471 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:39:06.752473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:06.752512 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:39:06.752606 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:06.753037 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:06.753051 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:06.753062 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 20:39:06.753299 (+ 237us) client_negotiation.cc:240] Negotiation successful
0806 20:39:06.753490 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":293,"threads_started":1}
I20260806 20:39:06.755999 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:44817
--builtin_ntp_servers=127.23.200.20:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 20:39:06.756168 2402 heartbeater.cc:347] Connected to a master server at 127.23.200.62:44817
I20260806 20:39:06.756480 2402 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:06.757085 2402 heartbeater.cc:511] Master 127.23.200.62:44817 requested a full tablet report, sending...
I20260806 20:39:06.759382 2209 ts_manager.cc:194] Registered new tserver with Master: 40b8181285da43efb7d837cabff05bf0 (127.23.200.1:45055)
W20260806 20:39:07.087169 2406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:07.087456 2406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:07.087559 2406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:07.094792 2406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:07.094892 2406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:07.095003 2406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:39:07.103960 2406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:44817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:07.105225 2406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:07.107012 2406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:07.120494 2415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:07.120530 2412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:07.120579 2413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:07.123761 2406 server_base.cc:1034] running on GCE node
I20260806 20:39:07.124392 2406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:07.125389 2406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:07.126622 2406 hybrid_clock.cc:648] HybridClock initialized: now 1786048747126560 us; error 58 us; skew 500 ppm
I20260806 20:39:07.126947 2406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:07.129206 2406 webserver.cc:460] Webserver started at http://127.23.200.2:38015/ using document root <none> and password file <none>
I20260806 20:39:07.129941 2406 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:07.130082 2406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:07.130512 2406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:07.133517 2406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "e015ac0584bb4114a6f2b03fec6c8032"
format_stamp: "Formatted at 2026-08-06 20:39:07 on dist-test-slave-tm1g"
server_key: "eefb2ee945183afb1ce298432f5582d8"
server_key_iv: "db1b0d433d5838c73c5eb8b927b55377"
server_key_version: "encryptionkey@0"
I20260806 20:39:07.134413 2406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "e015ac0584bb4114a6f2b03fec6c8032"
format_stamp: "Formatted at 2026-08-06 20:39:07 on dist-test-slave-tm1g"
server_key: "eefb2ee945183afb1ce298432f5582d8"
server_key_iv: "db1b0d433d5838c73c5eb8b927b55377"
server_key_version: "encryptionkey@0"
I20260806 20:39:07.139715 2406 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:39:07.143235 2421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:07.144595 2406 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:39:07.144780 2406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "e015ac0584bb4114a6f2b03fec6c8032"
format_stamp: "Formatted at 2026-08-06 20:39:07 on dist-test-slave-tm1g"
server_key: "eefb2ee945183afb1ce298432f5582d8"
server_key_iv: "db1b0d433d5838c73c5eb8b927b55377"
server_key_version: "encryptionkey@0"
I20260806 20:39:07.145005 2406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:07.150969 2406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:07.151782 2406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:07.152070 2406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:07.153024 2406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:07.154534 2406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:07.154657 2406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:07.154798 2406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:07.154891 2406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:07.190037 2406 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:43339
I20260806 20:39:07.190085 2534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:43339 every 8 connection(s)
I20260806 20:39:07.191426 2406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:39:07.200817 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.194199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:56511 (local address 127.23.200.62:44817)
0806 20:39:07.194448 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:07.194454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:07.196074 (+ 1620us) server_negotiation.cc:407] Connection header received
0806 20:39:07.198525 (+ 2451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:07.198529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:07.198585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:07.198666 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:07.200163 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:07.200168 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:07.200171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:07.200187 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:07.200228 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:07.200240 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:07.200335 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:07.200558 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 20:39:07.200611 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 20:39:07.201470 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2406
I20260806 20:39:07.201637 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:39:07.201834 2537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.194815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44817 (local address 127.23.200.2:56511)
0806 20:39:07.195894 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:07.195947 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:07.198188 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:07.198934 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:07.198961 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:07.199078 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:07.199795 (+ 717us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:39:07.199799 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:07.199849 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:39:07.199966 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:07.200344 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:07.200354 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:07.200362 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 20:39:07.200606 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 20:39:07.200798 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":311,"threads_started":1}
I20260806 20:39:07.202251 24352 external_mini_cluster.cc:1428] Setting key c4d104c36f3210d136c8b269057fa8f2
I20260806 20:39:07.203308 2535 heartbeater.cc:347] Connected to a master server at 127.23.200.62:44817
I20260806 20:39:07.203660 2535 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:07.204380 2535 heartbeater.cc:511] Master 127.23.200.62:44817 requested a full tablet report, sending...
I20260806 20:39:07.205401 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:44817
--builtin_ntp_servers=127.23.200.20:42345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 20:39:07.205816 2209 ts_manager.cc:194] Registered new tserver with Master: e015ac0584bb4114a6f2b03fec6c8032 (127.23.200.2:43339)
W20260806 20:39:07.538825 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:07.539108 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:07.539273 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:07.547047 2538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:07.547154 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:07.547363 2538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:39:07.556902 2538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:44817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:07.558401 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:07.560319 2538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:07.573693 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:07.573690 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:07.573931 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:07.574616 2538 server_base.cc:1034] running on GCE node
I20260806 20:39:07.575197 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:07.576159 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:07.577399 2538 hybrid_clock.cc:648] HybridClock initialized: now 1786048747577344 us; error 49 us; skew 500 ppm
I20260806 20:39:07.577729 2538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:39:07.580039 2538 webserver.cc:460] Webserver started at http://127.23.200.3:39743/ using document root <none> and password file <none>
I20260806 20:39:07.580757 2538 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:07.580895 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:07.581233 2538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:07.584179 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "8ee6b85e3a014bc3b4960c9a1b0369c9"
format_stamp: "Formatted at 2026-08-06 20:39:07 on dist-test-slave-tm1g"
server_key: "4a4569034146e190fc8517df46c8fee0"
server_key_iv: "4fcc2fa13ae359ad6ad4af25b7fa25f1"
server_key_version: "encryptionkey@0"
I20260806 20:39:07.584986 2538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "8ee6b85e3a014bc3b4960c9a1b0369c9"
format_stamp: "Formatted at 2026-08-06 20:39:07 on dist-test-slave-tm1g"
server_key: "4a4569034146e190fc8517df46c8fee0"
server_key_iv: "4fcc2fa13ae359ad6ad4af25b7fa25f1"
server_key_version: "encryptionkey@0"
I20260806 20:39:07.590035 2538 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:39:07.593482 2553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:07.594923 2538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:39:07.595098 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "8ee6b85e3a014bc3b4960c9a1b0369c9"
format_stamp: "Formatted at 2026-08-06 20:39:07 on dist-test-slave-tm1g"
server_key: "4a4569034146e190fc8517df46c8fee0"
server_key_iv: "4fcc2fa13ae359ad6ad4af25b7fa25f1"
server_key_version: "encryptionkey@0"
I20260806 20:39:07.595325 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:07.600981 2538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:07.601784 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:07.602067 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:07.603034 2538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:07.604487 2538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:07.604606 2538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:07.604748 2538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:07.604837 2538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:07.640698 2538 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:35517
I20260806 20:39:07.640759 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:35517 every 8 connection(s)
I20260806 20:39:07.642086 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:39:07.648931 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2538
I20260806 20:39:07.649137 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:39:07.649611 24352 external_mini_cluster.cc:1428] Setting key 606f43296b6ccbbad6af3df56ce2d4ca
I20260806 20:39:07.650563 2404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.644847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:47495 (local address 127.23.200.62:44817)
0806 20:39:07.645035 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:07.645039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:07.646474 (+ 1435us) server_negotiation.cc:407] Connection header received
0806 20:39:07.648114 (+ 1640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:07.648118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:07.648200 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:07.648287 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:07.649737 (+ 1450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:07.649743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:07.649748 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:07.649770 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:07.649809 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:07.649820 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:07.649974 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:07.650162 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 20:39:07.650239 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:39:07.651633 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.645386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44817 (local address 127.23.200.3:47495)
0806 20:39:07.646253 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:07.646293 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:07.647857 (+ 1564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:07.648549 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:07.648558 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:07.648647 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:07.649410 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:39:07.649413 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:07.649448 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:39:07.649566 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:07.649943 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:07.649953 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:07.649961 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 20:39:07.650248 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 20:39:07.650548 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":261,"threads_started":1}
I20260806 20:39:07.652978 2667 heartbeater.cc:347] Connected to a master server at 127.23.200.62:44817
I20260806 20:39:07.653234 2667 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:07.653754 2667 heartbeater.cc:511] Master 127.23.200.62:44817 requested a full tablet report, sending...
I20260806 20:39:07.655073 2209 ts_manager.cc:194] Registered new tserver with Master: 8ee6b85e3a014bc3b4960c9a1b0369c9 (127.23.200.3:35517)
I20260806 20:39:07.665118 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 20:39:07.671798 2404 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:39:07.669846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.23.200.62:44817)
0806 20:39:07.670057 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:07.670064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:07.670678 (+ 614us) server_negotiation.cc:407] Connection header received
0806 20:39:07.670869 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:07.670874 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:07.670934 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:07.671034 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:07.671503 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45452: BlockingRecv error: recv got EOF from 127.0.0.1:45452 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:39:07.672255 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2272
I20260806 20:39:07.676237 2678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.673385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39828 (local address 127.23.200.1:45055)
0806 20:39:07.673992 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:07.674000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:07.674028 (+ 28us) server_negotiation.cc:407] Connection header received
0806 20:39:07.674089 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:07.674096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:07.674411 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:07.674564 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:07.675560 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:07.675568 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:07.675581 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:07.675606 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:07.675659 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:07.675674 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:07.675788 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:07.675910 (+ 122us) server_negotiation.cc:299] Negotiation successful
0806 20:39:07.675990 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":304,"threads_started":1}
I20260806 20:39:07.677469 2397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:07.762346 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2272
I20260806 20:39:07.762591 2402 heartbeater.cc:503] Master 127.23.200.62:44817 was elected leader, sending a full tablet report...
I20260806 20:39:07.780985 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2406
I20260806 20:39:07.784821 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.782110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52568 (local address 127.23.200.2:43339)
0806 20:39:07.782685 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:07.782692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:07.782719 (+ 27us) server_negotiation.cc:407] Connection header received
0806 20:39:07.782782 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:07.782789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:07.783024 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:07.783154 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:07.784090 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:07.784097 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:07.784109 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:07.784129 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:07.784176 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:07.784190 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:07.784298 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:07.784384 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 20:39:07.784505 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":289,"threads_started":1}
I20260806 20:39:07.786125 2530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:07.867571 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2406
I20260806 20:39:07.884704 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2538
I20260806 20:39:07.888934 2682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:07.885697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46156 (local address 127.23.200.3:35517)
0806 20:39:07.886411 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:07.886419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:07.886451 (+ 32us) server_negotiation.cc:407] Connection header received
0806 20:39:07.886516 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:07.886522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:07.886751 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:07.886924 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:07.888144 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:07.888150 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:07.888161 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:39:07.888178 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:07.888217 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:39:07.888230 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:07.888434 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:07.888510 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 20:39:07.888613 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":325,"threads_started":1}
I20260806 20:39:07.890230 2662 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:07.972215 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2538
I20260806 20:39:07.989707 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2177
I20260806 20:39:07.991048 2242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:08.095885 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2177
2026-08-06T20:39:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2410 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:39:08 dist-test-slave-tm1g krb5kdc[2687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:39:08 dist-test-slave-tm1g krb5kdc[2687](info): set up 2 sockets
Aug 06 20:39:08 dist-test-slave-tm1g krb5kdc[2687](info): commencing operation
krb5kdc: starting...
W20260806 20:39:09.223124 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:39:09 dist-test-slave-tm1g krb5kdc[2687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:39:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:39:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:39:09.475988 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35673
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:39:09.810137 2701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:09.810451 2701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:09.810611 2701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:09.818011 2701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:39:09.818121 2701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:09.818205 2701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:09.818291 2701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:39:09.818382 2701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:39:09.828814 2701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:09.830235 2701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:09.831611 2701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:09.843559 2706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:09.843601 2707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:09.843590 2709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:09.844426 2701 server_base.cc:1034] running on GCE node
I20260806 20:39:09.845196 2701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:09.846688 2701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:09.847893 2701 hybrid_clock.cc:648] HybridClock initialized: now 1786048749847838 us; error 42 us; skew 500 ppm
I20260806 20:39:09.848253 2701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:39:09 dist-test-slave-tm1g krb5kdc[2687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048749, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:09.856540 2701 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:39:09.858217 2701 webserver.cc:460] Webserver started at http://127.23.200.62:44287/ using document root <none> and password file <none>
I20260806 20:39:09.859014 2701 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:09.859150 2701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:09.859488 2701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:09.863277 2701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "83619a50b7274a8f936a106033e6c6f8"
format_stamp: "Formatted at 2026-08-06 20:39:09 on dist-test-slave-tm1g"
server_key: "406a05302a2218c0dc3b5a63766a81f0"
server_key_iv: "9280a0712b49f6192eb7e8420c7388f0"
server_key_version: "encryptionkey@0"
I20260806 20:39:09.864380 2701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "83619a50b7274a8f936a106033e6c6f8"
format_stamp: "Formatted at 2026-08-06 20:39:09 on dist-test-slave-tm1g"
server_key: "406a05302a2218c0dc3b5a63766a81f0"
server_key_iv: "9280a0712b49f6192eb7e8420c7388f0"
server_key_version: "encryptionkey@0"
I20260806 20:39:09.871917 2701 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20260806 20:39:09.877133 2716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:09.879140 2701 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:39:09.879328 2701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "83619a50b7274a8f936a106033e6c6f8"
format_stamp: "Formatted at 2026-08-06 20:39:09 on dist-test-slave-tm1g"
server_key: "406a05302a2218c0dc3b5a63766a81f0"
server_key_iv: "9280a0712b49f6192eb7e8420c7388f0"
server_key_version: "encryptionkey@0"
I20260806 20:39:09.879591 2701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:09.901208 2701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:09.906167 2701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:09.906528 2701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:09.926520 2701 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:35673
I20260806 20:39:09.926553 2768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:35673 every 8 connection(s)
I20260806 20:39:09.927888 2701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:39:09.929754 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2701
I20260806 20:39:09.929993 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:39:09.930891 24352 external_mini_cluster.cc:1428] Setting key 6a402f1a000832eaf61170495c40abda
I20260806 20:39:09.932459 2769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:09.943775 2769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8: Bootstrap starting.
Aug 06 20:39:09 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:39:09.947402 2769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8: Neither blocks nor log segments found. Creating new log.
I20260806 20:39:09.948419 2769 log.cc:824] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:09.951653 2769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8: No bootstrap required, opened a new log
I20260806 20:39:09.952726 2774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:09.933036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43140 (local address 127.23.200.62:35673)
0806 20:39:09.933792 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:09.933805 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:09.933848 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:39:09.935465 (+ 1617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:09.935486 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:09.935851 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:09.936260 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:09.937792 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:09.939237 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:09.940514 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:09.941200 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:09.941413 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:09.945773 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:09.945780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:09.945797 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:09.945834 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:09.948913 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:09.949621 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:09.949630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:09.949639 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:09.949802 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:09.950213 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:09.950216 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:09.950218 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:09.950752 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:09.950976 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:09.951451 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 20:39:09.951800 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":351,"threads_started":1}
I20260806 20:39:09.959422 2769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83619a50b7274a8f936a106033e6c6f8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35673 } }
I20260806 20:39:09.959806 2769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:09.959921 2769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83619a50b7274a8f936a106033e6c6f8, State: Initialized, Role: FOLLOWER
I20260806 20:39:09.960466 2769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83619a50b7274a8f936a106033e6c6f8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35673 } }
I20260806 20:39:09.960705 2769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:39:09.960856 2769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:39:09.961030 2769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:09.963395 2769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83619a50b7274a8f936a106033e6c6f8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35673 } }
I20260806 20:39:09.963887 2769 leader_election.cc:302] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83619a50b7274a8f936a106033e6c6f8; no voters:
I20260806 20:39:09.964448 2769 leader_election.cc:288] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:39:09.964592 2776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:39:09.965313 2776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [term 1 LEADER]: Becoming Leader. State: Replica: 83619a50b7274a8f936a106033e6c6f8, State: Running, Role: LEADER
I20260806 20:39:09.965884 2776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83619a50b7274a8f936a106033e6c6f8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35673 } }
I20260806 20:39:09.966621 2769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:39:09.968602 2778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83619a50b7274a8f936a106033e6c6f8. Latest consensus state: current_term: 1 leader_uuid: "83619a50b7274a8f936a106033e6c6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83619a50b7274a8f936a106033e6c6f8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35673 } } }
I20260806 20:39:09.968992 2778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:09.968983 2777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83619a50b7274a8f936a106033e6c6f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83619a50b7274a8f936a106033e6c6f8" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35673 } } }
I20260806 20:39:09.969404 2777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8 [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:09.971848 2782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:39:09.977221 2782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:39:09.985864 2782 catalog_manager.cc:1347] Generated new cluster ID: ef06b8585d7d47248910bade03a51a41
I20260806 20:39:09.986111 2782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:39:09.999069 2782 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:39:10.000329 2782 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:39:10.008154 2782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8: Generated new TSK 0
I20260806 20:39:10.009112 2782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:39:10.074863 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35673
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:39:10.414855 2797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:10.415126 2797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:10.415220 2797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:10.422554 2797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:10.422658 2797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:10.422770 2797 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:39:10.432483 2797 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:10.433708 2797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:10.435025 2797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:10.448683 2802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:10.448783 2803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:10.448799 2805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:10.449721 2797 server_base.cc:1034] running on GCE node
I20260806 20:39:10.450397 2797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:10.451371 2797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:10.452636 2797 hybrid_clock.cc:648] HybridClock initialized: now 1786048750452576 us; error 42 us; skew 500 ppm
I20260806 20:39:10.453028 2797 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:39:10 dist-test-slave-tm1g krb5kdc[2687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:10.456339 2797 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:39:10.457751 2797 webserver.cc:460] Webserver started at http://127.23.200.1:41723/ using document root <none> and password file <none>
I20260806 20:39:10.458493 2797 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:10.458601 2797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:10.458889 2797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:10.461818 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "e9a11b52845043f2b7eb3b0227e70138"
format_stamp: "Formatted at 2026-08-06 20:39:10 on dist-test-slave-tm1g"
server_key: "975799409f74ad6dbf61d5f03b98c22a"
server_key_iv: "dd21c95382f9996a531ebc44f19a2664"
server_key_version: "encryptionkey@0"
I20260806 20:39:10.462603 2797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "e9a11b52845043f2b7eb3b0227e70138"
format_stamp: "Formatted at 2026-08-06 20:39:10 on dist-test-slave-tm1g"
server_key: "975799409f74ad6dbf61d5f03b98c22a"
server_key_iv: "dd21c95382f9996a531ebc44f19a2664"
server_key_version: "encryptionkey@0"
I20260806 20:39:10.468046 2797 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:39:10.471509 2812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:10.472913 2797 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 20:39:10.473045 2797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "e9a11b52845043f2b7eb3b0227e70138"
format_stamp: "Formatted at 2026-08-06 20:39:10 on dist-test-slave-tm1g"
server_key: "975799409f74ad6dbf61d5f03b98c22a"
server_key_iv: "dd21c95382f9996a531ebc44f19a2664"
server_key_version: "encryptionkey@0"
I20260806 20:39:10.473207 2797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:10.486471 2797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:10.490442 2797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:10.490700 2797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:10.491565 2797 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:10.492856 2797 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:10.492928 2797 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:10.493009 2797 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:10.493052 2797 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:10.525535 2797 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:45377
I20260806 20:39:10.525583 2928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:45377 every 8 connection(s)
I20260806 20:39:10.526939 2797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:39:10.528864 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2797
I20260806 20:39:10.529080 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:39:10.529966 24352 external_mini_cluster.cc:1428] Setting key bd7db36ab55e8747954bffda11b2e800
Aug 06 20:39:10 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:39:10.549896 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:10.529769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:47819 (local address 127.23.200.62:35673)
0806 20:39:10.530268 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:10.530273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:10.531484 (+ 1211us) server_negotiation.cc:407] Connection header received
0806 20:39:10.533620 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:10.533629 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:10.533729 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:10.533852 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:10.535472 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:10.536602 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:10.538596 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:10.539309 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:10.539470 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:10.543819 (+ 4349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:10.543825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:10.543828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:10.543863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:10.546651 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:10.547446 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:10.547450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:10.547452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:10.547586 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:10.548058 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:10.548063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:10.548068 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:10.548348 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:10.548521 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:10.549263 (+ 742us) server_negotiation.cc:299] Negotiation successful
0806 20:39:10.549520 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":284,"threads_started":1}
I20260806 20:39:10.551016 2932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:10.530440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35673 (local address 127.23.200.1:47819)
0806 20:39:10.531299 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:10.531359 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:10.533322 (+ 1963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:10.534058 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:10.534067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:10.534699 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:10.535275 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:10.535292 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:10.536754 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:10.536758 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:10.538441 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:10.538454 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:10.539532 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:10.539537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:10.539741 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:10.540507 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:10.540541 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:10.543551 (+ 3010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:10.546805 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:10.546815 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:10.546841 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:10.547297 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:10.547711 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:10.547716 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:10.547720 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:10.547923 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:10.548486 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:10.548495 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:10.548860 (+ 365us) client_negotiation.cc:769] Sending connection context
0806 20:39:10.549145 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 20:39:10.549722 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
I20260806 20:39:10.552842 2929 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35673
I20260806 20:39:10.553251 2929 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:10.554280 2929 heartbeater.cc:511] Master 127.23.200.62:35673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:39:10.556815 2733 ts_manager.cc:194] Registered new tserver with Master: e9a11b52845043f2b7eb3b0227e70138 (127.23.200.1:45377)
I20260806 20:39:10.558444 2733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:47819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:39:10.590225 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35673
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:39:10.926879 2935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:10.927210 2935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:10.927388 2935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:10.934894 2935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:10.935051 2935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:10.935227 2935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:39:10.945340 2935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:10.946797 2935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:10.948164 2935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:10.962023 2940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:10.962061 2943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:10.961975 2941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:10.962980 2935 server_base.cc:1034] running on GCE node
I20260806 20:39:10.963568 2935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:10.964532 2935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:10.965749 2935 hybrid_clock.cc:648] HybridClock initialized: now 1786048750965686 us; error 52 us; skew 500 ppm
I20260806 20:39:10.966097 2935 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:39:10 dist-test-slave-tm1g krb5kdc[2687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:10.969816 2935 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:39:10.971387 2935 webserver.cc:460] Webserver started at http://127.23.200.2:40735/ using document root <none> and password file <none>
I20260806 20:39:10.972126 2935 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:10.972280 2935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:10.972649 2935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:10.975694 2935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "f68c352ba9e6495f8cf695ee793fa8f5"
format_stamp: "Formatted at 2026-08-06 20:39:10 on dist-test-slave-tm1g"
server_key: "0fc0f197ebe028b3cc07892000c73364"
server_key_iv: "cf79b22573439d06aaed6745552286f9"
server_key_version: "encryptionkey@0"
I20260806 20:39:10.976516 2935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "f68c352ba9e6495f8cf695ee793fa8f5"
format_stamp: "Formatted at 2026-08-06 20:39:10 on dist-test-slave-tm1g"
server_key: "0fc0f197ebe028b3cc07892000c73364"
server_key_iv: "cf79b22573439d06aaed6745552286f9"
server_key_version: "encryptionkey@0"
I20260806 20:39:10.981899 2935 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:39:10.985638 2950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:10.987242 2935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 20:39:10.987419 2935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "f68c352ba9e6495f8cf695ee793fa8f5"
format_stamp: "Formatted at 2026-08-06 20:39:10 on dist-test-slave-tm1g"
server_key: "0fc0f197ebe028b3cc07892000c73364"
server_key_iv: "cf79b22573439d06aaed6745552286f9"
server_key_version: "encryptionkey@0"
I20260806 20:39:10.987613 2935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:11.004659 2935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:11.008873 2935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:11.009182 2935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:11.010162 2935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:11.011574 2935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:11.011646 2935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:11.011817 2935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:11.011870 2935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:11.045833 2935 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:46037
I20260806 20:39:11.045923 3066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:46037 every 8 connection(s)
I20260806 20:39:11.047204 2935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:39:11.055150 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 2935
I20260806 20:39:11.055334 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:39:11.055769 24352 external_mini_cluster.cc:1428] Setting key 25eadbbdc1ca0299e62da30a2aed194e
Aug 06 20:39:11 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048750, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:39:11.067987 2931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.049629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:58813 (local address 127.23.200.62:35673)
0806 20:39:11.049809 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.049815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.050883 (+ 1068us) server_negotiation.cc:407] Connection header received
0806 20:39:11.052660 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.052666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.052759 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.052872 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:11.054332 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.055092 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.056933 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.057374 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.057544 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.061006 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:11.061013 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:11.061020 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:11.061066 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:11.064671 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.065372 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.065380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.065386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.065555 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.065961 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.065969 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.065974 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.066293 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:11.066467 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.067491 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.067738 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 20:39:11.069512 3069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.049987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35673 (local address 127.23.200.2:58813)
0806 20:39:11.050724 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.050760 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.052392 (+ 1632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.053104 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.053112 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.053593 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:11.054140 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.054157 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.055251 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.055257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.056786 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.056803 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.057652 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.057660 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.057920 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.058655 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:11.058684 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:11.060776 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:11.064817 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:11.064825 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:11.064837 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:11.065245 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:11.065667 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:11.065670 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:11.065672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:11.065844 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:11.066617 (+ 773us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:11.066626 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:11.067096 (+ 470us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.067435 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.068042 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":323,"threads_started":1}
I20260806 20:39:11.071014 3067 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35673
I20260806 20:39:11.071354 3067 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:11.072115 3067 heartbeater.cc:511] Master 127.23.200.62:35673 requested a full tablet report, sending...
I20260806 20:39:11.073757 2733 ts_manager.cc:194] Registered new tserver with Master: f68c352ba9e6495f8cf695ee793fa8f5 (127.23.200.2:46037)
I20260806 20:39:11.075069 2733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:58813
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:39:11.113251 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35673
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:39:11.449128 3072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:11.449417 3072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:11.449517 3072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:11.457262 3072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:11.457379 3072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:11.457494 3072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:39:11.467687 3072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:11.469043 3072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:11.470430 3072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:11.484056 3080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:11.484102 3078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:11.484338 3077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:11.485216 3072 server_base.cc:1034] running on GCE node
I20260806 20:39:11.485886 3072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:11.486936 3072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:11.488180 3072 hybrid_clock.cc:648] HybridClock initialized: now 1786048751488122 us; error 53 us; skew 500 ppm
I20260806 20:39:11.488515 3072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:39:11 dist-test-slave-tm1g krb5kdc[2687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048751, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:11.491834 3072 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:39:11.493438 3072 webserver.cc:460] Webserver started at http://127.23.200.3:37427/ using document root <none> and password file <none>
I20260806 20:39:11.494135 3072 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:11.494236 3072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:11.494577 3072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:39:11.497540 3072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "c18b76c26ffc41a682a6e9874e3e6289"
format_stamp: "Formatted at 2026-08-06 20:39:11 on dist-test-slave-tm1g"
server_key: "94b8f6249ad4fbd6def84451adce836a"
server_key_iv: "d137bc97ea92f05887b910467f8f270b"
server_key_version: "encryptionkey@0"
I20260806 20:39:11.498260 3072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "c18b76c26ffc41a682a6e9874e3e6289"
format_stamp: "Formatted at 2026-08-06 20:39:11 on dist-test-slave-tm1g"
server_key: "94b8f6249ad4fbd6def84451adce836a"
server_key_iv: "d137bc97ea92f05887b910467f8f270b"
server_key_version: "encryptionkey@0"
I20260806 20:39:11.503893 3072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260806 20:39:11.507409 3088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:11.508775 3072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:39:11.508912 3072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "c18b76c26ffc41a682a6e9874e3e6289"
format_stamp: "Formatted at 2026-08-06 20:39:11 on dist-test-slave-tm1g"
server_key: "94b8f6249ad4fbd6def84451adce836a"
server_key_iv: "d137bc97ea92f05887b910467f8f270b"
server_key_version: "encryptionkey@0"
I20260806 20:39:11.509088 3072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:11.522858 3072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:11.527451 3072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:11.527772 3072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:11.528777 3072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:11.530263 3072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:11.530401 3072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 20:39:11.530558 3072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:11.530637 3072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:11.561597 2929 heartbeater.cc:503] Master 127.23.200.62:35673 was elected leader, sending a full tablet report...
I20260806 20:39:11.565088 3072 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:44227
I20260806 20:39:11.565135 3202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:44227 every 8 connection(s)
I20260806 20:39:11.566460 3072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:39:11.567521 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 3072
I20260806 20:39:11.567734 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:39:11.568361 24352 external_mini_cluster.cc:1428] Setting key be92dc0eb0fed1fcf4d26e7b87e4a940
Aug 06 20:39:11 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048751, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:39:11.587329 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.569440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:35175 (local address 127.23.200.62:35673)
0806 20:39:11.569971 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.569978 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.571250 (+ 1272us) server_negotiation.cc:407] Connection header received
0806 20:39:11.573067 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.573071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.573143 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.573230 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:11.574650 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.575412 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.576950 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.577479 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.577575 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.581042 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:11.581047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:11.581051 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:11.581086 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:11.584323 (+ 3237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.585100 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.585104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.585106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.585230 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.585676 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.585679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.585681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.585945 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:11.586143 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.586868 (+ 725us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.587039 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":289,"threads_started":1}
I20260806 20:39:11.588254 3206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.570131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35673 (local address 127.23.200.3:35175)
0806 20:39:11.571060 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.571108 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.572782 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.573396 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.573404 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.573887 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:11.574507 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.574526 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.575560 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.575564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.576822 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.576833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.577662 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.577666 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.577856 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.578573 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:11.578595 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:11.580831 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:11.584459 (+ 3628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:11.584467 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:11.584480 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:11.584977 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:11.585342 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:11.585345 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:11.585348 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:11.585561 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:11.586126 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:11.586132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:11.586511 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.586776 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.587169 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":414,"threads_started":1}
I20260806 20:39:11.589564 3203 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35673
I20260806 20:39:11.589838 3203 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:11.590618 3203 heartbeater.cc:511] Master 127.23.200.62:35673 requested a full tablet report, sending...
I20260806 20:39:11.592033 2733 ts_manager.cc:194] Registered new tserver with Master: c18b76c26ffc41a682a6e9874e3e6289 (127.23.200.3:44227)
I20260806 20:39:11.592886 2733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:35175
I20260806 20:39:11.596189 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:39:11.612973 3205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.600570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43156 (local address 127.23.200.62:35673)
0806 20:39:11.600796 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.600805 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.601252 (+ 447us) server_negotiation.cc:407] Connection header received
0806 20:39:11.601388 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.601392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.601451 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.601535 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:11.603004 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.603730 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.604929 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.605335 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.605425 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.607697 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:11.607702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:11.607706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:11.607736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:11.610144 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.610839 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.610843 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.610845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.610971 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.611478 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.611482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.611484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.611856 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:11.612055 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.612413 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.612592 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:39:11.617195 2733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:39:11.624318 2733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:39:11.662714 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.637173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:44227 (local address 127.0.0.1:38300)
0806 20:39:11.649361 (+ 12188us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.649430 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.649712 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.650971 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.650981 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.651022 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:39:11.651782 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.651816 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.654649 (+ 2833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.654657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.658093 (+ 3436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.658115 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.661219 (+ 3104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.661228 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.661529 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.661612 (+ 83us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.661810 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.662102 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11772,"thread_start_us":13199,"threads_started":1}
I20260806 20:39:11.663820 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.637272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38300 (local address 127.23.200.3:44227)
0806 20:39:11.650043 (+ 12771us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.650057 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.650094 (+ 37us) server_negotiation.cc:407] Connection header received
0806 20:39:11.650201 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.650210 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.650532 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.650759 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:39:11.652120 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.654392 (+ 2272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.658370 (+ 3978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.660481 (+ 2111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.660625 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.660982 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.663088 (+ 2106us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.663330 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12552,"thread_start_us":13177,"threads_started":1}
I20260806 20:39:11.663098 3216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.637186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:46037 (local address 127.0.0.1:56982)
0806 20:39:11.651460 (+ 14274us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.651481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.651660 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.652581 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.652589 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.652626 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:39:11.653167 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.653183 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.655994 (+ 2811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.656002 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.659086 (+ 3084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.659103 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.661784 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.661795 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.662096 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.662152 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.662235 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.662441 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14132,"mutex_wait_us":3,"thread_start_us":13641,"threads_started":1}
I20260806 20:39:11.664646 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.637733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56982 (local address 127.23.200.2:46037)
0806 20:39:11.650347 (+ 12614us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.650356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.651863 (+ 1507us) server_negotiation.cc:407] Connection header received
0806 20:39:11.651988 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.651999 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.652313 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.653407 (+ 1094us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:39:11.653692 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.655777 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.659380 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.661465 (+ 2085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.663292 (+ 1827us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.663605 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.663706 (+ 101us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.664052 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12445,"thread_start_us":12341,"threads_started":1}
I20260806 20:39:11.668607 3137 tablet_service.cc:1467] Processing CreateTablet for tablet ec7cf9e4206642d2a2616485f31d5f3d (DEFAULT_TABLE table=test-table [id=36af13d6a843456f9ff4e0744e7f4a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:39:11.670214 3137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7cf9e4206642d2a2616485f31d5f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:11.669800 3000 tablet_service.cc:1467] Processing CreateTablet for tablet ec7cf9e4206642d2a2616485f31d5f3d (DEFAULT_TABLE table=test-table [id=36af13d6a843456f9ff4e0744e7f4a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:39:11.671272 3000 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7cf9e4206642d2a2616485f31d5f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:11.673578 3217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.651473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45377 (local address 127.0.0.1:60906)
0806 20:39:11.664571 (+ 13098us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.664596 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.664744 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.665920 (+ 1176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.665925 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.665953 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:39:11.666140 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.666150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.668171 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.668177 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.670001 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.670017 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.672725 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.672731 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.672931 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.673008 (+ 77us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.673105 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.673253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12948,"thread_start_us":14045,"threads_started":1}
I20260806 20:39:11.674337 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.651843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60906 (local address 127.23.200.1:45377)
0806 20:39:11.665167 (+ 13324us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.665181 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.665220 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:39:11.665374 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.665386 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.665680 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.665913 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:39:11.666327 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.668005 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.670207 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.672232 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.672387 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.672622 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.673812 (+ 1190us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.673998 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12331,"thread_start_us":13140,"threads_started":1}
I20260806 20:39:11.678642 2862 tablet_service.cc:1467] Processing CreateTablet for tablet ec7cf9e4206642d2a2616485f31d5f3d (DEFAULT_TABLE table=test-table [id=36af13d6a843456f9ff4e0744e7f4a0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:39:11.680265 2862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec7cf9e4206642d2a2616485f31d5f3d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:11.684679 3222 tablet_bootstrap.cc:492] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Bootstrap starting.
I20260806 20:39:11.686079 3223 tablet_bootstrap.cc:492] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Bootstrap starting.
I20260806 20:39:11.688694 3222 tablet_bootstrap.cc:654] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Neither blocks nor log segments found. Creating new log.
I20260806 20:39:11.689785 3223 tablet_bootstrap.cc:654] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Neither blocks nor log segments found. Creating new log.
I20260806 20:39:11.690068 3222 log.cc:824] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:11.691190 3223 log.cc:824] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:11.693673 3222 tablet_bootstrap.cc:492] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: No bootstrap required, opened a new log
I20260806 20:39:11.694010 3222 ts_tablet_manager.cc:1397] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260806 20:39:11.694458 3223 tablet_bootstrap.cc:492] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: No bootstrap required, opened a new log
I20260806 20:39:11.694722 3223 ts_tablet_manager.cc:1397] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20260806 20:39:11.694916 3227 tablet_bootstrap.cc:492] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Bootstrap starting.
I20260806 20:39:11.699723 3227 tablet_bootstrap.cc:654] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Neither blocks nor log segments found. Creating new log.
I20260806 20:39:11.701064 3227 log.cc:824] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:11.703094 3223 raft_consensus.cc:348] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.703483 3223 raft_consensus.cc:374] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:11.703586 3223 raft_consensus.cc:729] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c18b76c26ffc41a682a6e9874e3e6289, State: Initialized, Role: FOLLOWER
I20260806 20:39:11.704227 3223 consensus_queue.cc:260] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.704391 3227 tablet_bootstrap.cc:492] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: No bootstrap required, opened a new log
I20260806 20:39:11.704672 3227 ts_tablet_manager.cc:1397] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260806 20:39:11.704468 3222 raft_consensus.cc:348] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.704928 3222 raft_consensus.cc:374] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:11.705053 3222 raft_consensus.cc:729] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f68c352ba9e6495f8cf695ee793fa8f5, State: Initialized, Role: FOLLOWER
I20260806 20:39:11.705611 3203 heartbeater.cc:503] Master 127.23.200.62:35673 was elected leader, sending a full tablet report...
I20260806 20:39:11.705778 3222 consensus_queue.cc:260] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.706108 3223 ts_tablet_manager.cc:1428] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 20:39:11.707038 3067 heartbeater.cc:503] Master 127.23.200.62:35673 was elected leader, sending a full tablet report...
I20260806 20:39:11.707733 3222 ts_tablet_manager.cc:1428] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Time spent starting tablet: real 0.014s user 0.004s sys 0.010s
I20260806 20:39:11.715353 3227 raft_consensus.cc:348] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.715730 3227 raft_consensus.cc:374] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:11.715806 3227 raft_consensus.cc:729] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9a11b52845043f2b7eb3b0227e70138, State: Initialized, Role: FOLLOWER
I20260806 20:39:11.716305 3227 consensus_queue.cc:260] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.718004 3227 ts_tablet_manager.cc:1428] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260806 20:39:11.741875 3231 raft_consensus.cc:482] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:39:11.742275 3231 raft_consensus.cc:504] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.744766 3231 leader_election.cc:288] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c18b76c26ffc41a682a6e9874e3e6289 (127.23.200.3:44227), f68c352ba9e6495f8cf695ee793fa8f5 (127.23.200.2:46037)
I20260806 20:39:11.753163 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.746418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:36879 (local address 127.23.200.3:44227)
0806 20:39:11.746673 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.746682 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.746708 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:39:11.746800 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.746809 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.746909 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.747029 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:39:11.747802 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.748944 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.751204 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.752200 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.752317 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.752370 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.752761 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.752861 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:39:11.753242 3232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.745030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:44227 (local address 127.23.200.1:36879)
0806 20:39:11.745971 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.745999 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.746178 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.747063 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.747066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.747094 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:39:11.747644 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.747656 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.749084 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.749090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.751045 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.751058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.752408 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.752416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.752583 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.752624 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.752723 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.752851 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":800,"thread_start_us":583,"threads_started":1}
I20260806 20:39:11.753813 3233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.745229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:46037 (local address 127.23.200.1:40819)
0806 20:39:11.747778 (+ 2549us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:11.747796 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:11.747923 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:11.748209 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:11.748215 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:11.748240 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:39:11.748558 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.748568 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.749852 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.749857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.751655 (+ 1798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:11.751670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.753053 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.753059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:11.753233 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.753271 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:39:11.753440 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 20:39:11.753559 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2395,"thread_start_us":2131,"threads_started":1}
I20260806 20:39:11.753902 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.745347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:40819 (local address 127.23.200.2:46037)
0806 20:39:11.746011 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.746020 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.747868 (+ 1848us) server_negotiation.cc:407] Connection header received
0806 20:39:11.748035 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.748039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.748098 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.748159 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:39:11.748715 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.749698 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.751814 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.752819 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.752963 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.753044 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.753499 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.753655 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498}
I20260806 20:39:11.754467 3157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7cf9e4206642d2a2616485f31d5f3d" candidate_uuid: "e9a11b52845043f2b7eb3b0227e70138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18b76c26ffc41a682a6e9874e3e6289" is_pre_election: true
I20260806 20:39:11.754985 3157 raft_consensus.cc:2454] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a11b52845043f2b7eb3b0227e70138 in term 0.
I20260806 20:39:11.755687 2814 leader_election.cc:302] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18b76c26ffc41a682a6e9874e3e6289, e9a11b52845043f2b7eb3b0227e70138; no voters:
I20260806 20:39:11.756155 3231 raft_consensus.cc:2781] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:39:11.756330 3231 raft_consensus.cc:482] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:39:11.756498 3231 raft_consensus.cc:3037] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:11.756364 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7cf9e4206642d2a2616485f31d5f3d" candidate_uuid: "e9a11b52845043f2b7eb3b0227e70138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" is_pre_election: true
I20260806 20:39:11.756932 3021 raft_consensus.cc:2454] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9a11b52845043f2b7eb3b0227e70138 in term 0.
I20260806 20:39:11.759189 3231 raft_consensus.cc:504] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.759768 3231 leader_election.cc:288] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [CANDIDATE]: Term 1 election: Requested vote from peers c18b76c26ffc41a682a6e9874e3e6289 (127.23.200.3:44227), f68c352ba9e6495f8cf695ee793fa8f5 (127.23.200.2:46037)
I20260806 20:39:11.760229 3157 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7cf9e4206642d2a2616485f31d5f3d" candidate_uuid: "e9a11b52845043f2b7eb3b0227e70138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c18b76c26ffc41a682a6e9874e3e6289"
I20260806 20:39:11.760264 3021 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec7cf9e4206642d2a2616485f31d5f3d" candidate_uuid: "e9a11b52845043f2b7eb3b0227e70138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f68c352ba9e6495f8cf695ee793fa8f5"
I20260806 20:39:11.760429 3157 raft_consensus.cc:3037] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:11.760509 3021 raft_consensus.cc:3037] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:11.762755 3157 raft_consensus.cc:2454] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a11b52845043f2b7eb3b0227e70138 in term 1.
I20260806 20:39:11.762897 3021 raft_consensus.cc:2454] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9a11b52845043f2b7eb3b0227e70138 in term 1.
I20260806 20:39:11.763188 2814 leader_election.cc:302] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c18b76c26ffc41a682a6e9874e3e6289, e9a11b52845043f2b7eb3b0227e70138; no voters:
I20260806 20:39:11.763499 3231 raft_consensus.cc:2781] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:39:11.763909 3231 raft_consensus.cc:686] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 1 LEADER]: Becoming Leader. State: Replica: e9a11b52845043f2b7eb3b0227e70138, State: Running, Role: LEADER
I20260806 20:39:11.764391 3231 consensus_queue.cc:237] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } }
I20260806 20:39:11.769460 2733 catalog_manager.cc:5495] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9a11b52845043f2b7eb3b0227e70138 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "e9a11b52845043f2b7eb3b0227e70138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9a11b52845043f2b7eb3b0227e70138" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45377 } health_report { overall_health: HEALTHY } } }
W20260806 20:39:11.779290 2930 tablet.cc:2369] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:39:11.799520 3068 tablet.cc:2369] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:39:11.807113 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.801610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60920 (local address 127.23.200.1:45377)
0806 20:39:11.801837 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.801844 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.801865 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:39:11.802075 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.802079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.802160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.802341 (+ 181us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:39:11.802735 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.803495 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.804832 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.805263 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.805421 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.805918 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:39:11.806517 (+ 599us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:39:11.806702 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.806754 (+ 52us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.806839 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 20:39:11.815896 3157 raft_consensus.cc:1261] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 1 FOLLOWER]: Refusing update from remote peer e9a11b52845043f2b7eb3b0227e70138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:39:11.815893 3021 raft_consensus.cc:1261] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 1 FOLLOWER]: Refusing update from remote peer e9a11b52845043f2b7eb3b0227e70138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:39:11.816915 3234 consensus_queue.cc:1035] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f68c352ba9e6495f8cf695ee793fa8f5" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 46037 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:39:11.817262 3231 consensus_queue.cc:1035] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c18b76c26ffc41a682a6e9874e3e6289" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 44227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:39:11.818143 3204 tablet.cc:2369] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:39:11.829520 3237 mvcc.cc:205] Tried to move back new op lower bound from 7315655687426572288 to 7315655687230619648. Current Snapshot: MvccSnapshot[applied={T|T < 7315655687426572288}]
I20260806 20:39:11.845983 2732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:39:11.846227 2732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:39:11.849977 2732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 83619a50b7274a8f936a106033e6c6f8: Sending DeleteTablet for 3 replicas of tablet ec7cf9e4206642d2a2616485f31d5f3d
I20260806 20:39:11.851295 3137 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7cf9e4206642d2a2616485f31d5f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:39:11 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38300
I20260806 20:39:11.851477 2862 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7cf9e4206642d2a2616485f31d5f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:39:11 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:60906
I20260806 20:39:11.852567 3000 tablet_service.cc:1514] Processing DeleteTablet for tablet ec7cf9e4206642d2a2616485f31d5f3d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:39:11 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56982
I20260806 20:39:11.852738 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2797
I20260806 20:39:11.862882 3248 tablet_replica.cc:330] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: stopping tablet replica
I20260806 20:39:11.863502 3248 raft_consensus.cc:2229] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:39:11.864439 3248 raft_consensus.cc:2258] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:39:11.866958 3247 tablet_replica.cc:330] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: stopping tablet replica
I20260806 20:39:11.867274 3248 ts_tablet_manager.cc:1905] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:39:11.867511 3247 raft_consensus.cc:2229] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:39:11.868117 3247 raft_consensus.cc:2258] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:39:11.870826 3247 ts_tablet_manager.cc:1905] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:39:11.876219 3249 tablet_replica.cc:330] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: stopping tablet replica
I20260806 20:39:11.876845 3249 raft_consensus.cc:2229] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:39:11.877358 3249 raft_consensus.cc:2258] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:39:11.878752 3247 ts_tablet_manager.cc:1918] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:39:11.878937 3247 log.cc:1196] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/ec7cf9e4206642d2a2616485f31d5f3d
I20260806 20:39:11.879526 3247 ts_tablet_manager.cc:1939] T ec7cf9e4206642d2a2616485f31d5f3d P c18b76c26ffc41a682a6e9874e3e6289: Deleting consensus metadata
I20260806 20:39:11.880239 3249 ts_tablet_manager.cc:1905] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:39:11.880463 3248 ts_tablet_manager.cc:1918] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:39:11.880648 3248 log.cc:1196] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/ec7cf9e4206642d2a2616485f31d5f3d
I20260806 20:39:11.881206 3248 ts_tablet_manager.cc:1939] T ec7cf9e4206642d2a2616485f31d5f3d P e9a11b52845043f2b7eb3b0227e70138: Deleting consensus metadata
I20260806 20:39:11.881529 2718 catalog_manager.cc:4841] TS c18b76c26ffc41a682a6e9874e3e6289 (127.23.200.3:44227): tablet ec7cf9e4206642d2a2616485f31d5f3d (table test-table [id=36af13d6a843456f9ff4e0744e7f4a0e]) successfully deleted
I20260806 20:39:11.883006 2719 catalog_manager.cc:4841] TS e9a11b52845043f2b7eb3b0227e70138 (127.23.200.1:45377): tablet ec7cf9e4206642d2a2616485f31d5f3d (table test-table [id=36af13d6a843456f9ff4e0744e7f4a0e]) successfully deleted
Aug 06 20:39:11 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:39:11.887836 3249 ts_tablet_manager.cc:1918] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:39:11.888053 3249 log.cc:1196] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/ec7cf9e4206642d2a2616485f31d5f3d
I20260806 20:39:11.888645 3249 ts_tablet_manager.cc:1939] T ec7cf9e4206642d2a2616485f31d5f3d P f68c352ba9e6495f8cf695ee793fa8f5: Deleting consensus metadata
I20260806 20:39:11.890579 2719 catalog_manager.cc:4841] TS f68c352ba9e6495f8cf695ee793fa8f5 (127.23.200.2:46037): tablet ec7cf9e4206642d2a2616485f31d5f3d (table test-table [id=36af13d6a843456f9ff4e0744e7f4a0e]) successfully deleted
I20260806 20:39:11.890836 3221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:11.854532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.23.200.1:45377)
0806 20:39:11.854717 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:11.854723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:11.868589 (+ 13866us) server_negotiation.cc:407] Connection header received
0806 20:39:11.868877 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:11.868898 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:11.868986 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:11.869074 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:11.870878 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.872084 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.874030 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:11.874777 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:11.874874 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:11.884265 (+ 9391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:11.884277 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:11.884292 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:11.884332 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:11.887706 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.888532 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.888540 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.888542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.888730 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:11.889154 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:11.889161 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:11.889168 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:11.889534 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:11.889852 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:11.890157 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 20:39:11.890442 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:39:11.891881 2924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:11.989549 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2797
I20260806 20:39:12.010093 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2935
Aug 06 20:39:12 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:39:12.025995 3219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:12.011088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.23.200.2:46037)
0806 20:39:12.011293 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:12.011299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:12.011320 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:39:12.011433 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:12.011438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:12.011520 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:12.011633 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:12.013109 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:12.013907 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:12.015245 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:12.015646 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:12.015785 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:12.019984 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:12.020000 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:12.020018 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:12.020060 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:12.023151 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:12.023845 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:12.023853 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:12.023856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:12.023984 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:12.024375 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:12.024380 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:12.024383 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:12.024780 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:12.025093 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:12.025415 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 20:39:12.025652 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:39:12.026719 3062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:12.116063 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2935
I20260806 20:39:12.134367 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 3072
Aug 06 20:39:12 dist-test-slave-tm1g krb5kdc[2687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048749, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:39:12.151283 3218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:12.135197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38308 (local address 127.23.200.3:44227)
0806 20:39:12.135399 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:12.135406 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:12.135488 (+ 82us) server_negotiation.cc:407] Connection header received
0806 20:39:12.135707 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:12.135713 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:12.135791 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:12.135917 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:12.137220 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:12.138221 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:12.139539 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:12.139954 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:12.140087 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:12.144960 (+ 4873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:12.144976 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:12.145000 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:12.145042 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:12.148360 (+ 3318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:12.149075 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:12.149083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:12.149086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:12.149277 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:12.149711 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:12.149716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:12.149718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:12.150126 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:12.150477 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:12.150783 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 20:39:12.150991 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:39:12.152096 3198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:12.245405 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 3072
I20260806 20:39:12.263624 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 2701
I20260806 20:39:12.264915 2764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:39:12.377630 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 2701
2026-08-06T20:39:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4283 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:39:12 dist-test-slave-tm1g krb5kdc[3257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:39:12 dist-test-slave-tm1g krb5kdc[3257](info): set up 2 sockets
Aug 06 20:39:12 dist-test-slave-tm1g krb5kdc[3257](info): commencing operation
krb5kdc: starting...
W20260806 20:39:13.510942 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:39:13 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048753, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:39:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:39:13Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/rangeradmin_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/rangeradmin_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/rangerlookup_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/rangerlookup_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
I20260806 20:39:13.874423 24352 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskG_AEDx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/postgres -l logfile start
2026-08-06 20:39:16.458 UTC [3288] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 20:39:16.459 UTC [3288] LOG: listening on IPv4 address "127.23.200.20", port 38289
2026-08-06 20:39:16.462 UTC [3288] LOG: listening on Unix socket "/tmp/.s.PGSQL.38289"
2026-08-06 20:39:16.485 UTC [3291] LOG: database system was shut down at 2026-08-06 20:39:15 UTC
2026-08-06 20:39:16.488 UTC [3288] LOG: database system is ready to accept connections
I20260806 20:39:17.502625 24352 mini_postgres.cc:96] Postgres bound to 38289
2026-08-06 20:39:17.522 UTC [3300] FATAL: database "slave" does not exist
127.23.200.20:38289 - accepting connections
I20260806 20:39:17.523708 24352 mini_ranger.cc:161] Starting Ranger...
I20260806 20:39:17.552166 24352 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 20:39:18.053670 24352 mini_ranger.cc:87] Created ranger Postgres database
I20260806 20:39:18.053818 24352 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-admin
2026-08-06 20:39:18,430 [I] DB FLAVOR :POSTGRES
2026-08-06 20:39:18,431 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 20:39:18,431 [I] Checking connection..
2026-08-06 20:39:18,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 20:39:18,722 [I] Checking connection passed.
2026-08-06 20:39:18,723 [I] --------- Verifying version history table ---------
2026-08-06 20:39:18,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 20:39:18,976 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 20:39:18,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 20:39:19,233 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 20:39:19,233 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 20:39:19,233 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 20:39:19.498 UTC [3371] WARNING: there is no transaction in progress
2026-08-06 20:39:19,505 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 20:39:19,505 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 20:39:19,775 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 20:39:19,776 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 20:39:19,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 20:39:20,041 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 20:39:20,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tm1g.c.gcp-upstream.internal','N') ;"
2026-08-06 20:39:20,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 20:39:20,800 [I] Table x_portal_user does not exist in database ranger
2026-08-06 20:39:20,800 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 20:39:21,069 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 20:39:21,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 20:39:21,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 20:39:21,564 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 20:39:21,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 20:39:22.554 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.570 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.585 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.600 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.615 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.634 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.689 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.696 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.711 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.721 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.731 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.744 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.754 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.760 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.767 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:22.774 UTC [3486] WARNING: there is no transaction in progress
2026-08-06 20:39:23,405 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 20:39:23,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 20:39:23,662 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 20:39:23,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 20:39:23,927 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 20:39:23,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 20:39:24,181 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 20:39:24,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 20:39:24,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 20:39:24,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tm1g.c.gcp-upstream.internal';"
2026-08-06 20:39:24,927 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 20:39:24,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 20:39:25,190 [I] DB_PATCHES have already been applied
I20260806 20:39:25.196215 24352 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-admin:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskG_AEDx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 20:39:25.196363 24352 mini_ranger.cc:193] Using host: 127.23.200.20
I20260806 20:39:25.213150 24352 mini_ranger.cc:227] Ranger admin URL: http://127.23.200.20:52497
Aug 06, 2026 8:39:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 8:39:25 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 8:39:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 8:39:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 8:39:25 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 8:39:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 8:39:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 20:39:26.358917 24352 mini_ranger.cc:160] Time spent starting Ranger: real 8.835s user 0.000s sys 0.044s
Aug 06, 2026 8:39:47 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 8:39:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 8:39:47 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 8:39:47 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 8:39:48 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 20:39:48.658804 24352 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/rangerkms_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/rangerkms_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.200.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.200.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 20:39:48 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048788, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 20:39:48.833520 24352 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 20:39:48.856104 24352 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 20:39:49.645131 24352 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 20:39:49.645277 24352 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms
I20260806 20:39:49.645462 24352 mini_ranger_kms.cc:227] Using postgres at 127.23.200.20:38289
2026-08-06 20:39:50,133 [I] DB FLAVOR :POSTGRES
2026-08-06 20:39:50,134 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 20:39:50,134 [I] Checking connection
2026-08-06 20:39:50,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 20:39:50,391 [I] connection success
2026-08-06 20:39:50,391 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 20:39:50,391 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 20:39:50,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 20:39:50,642 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 20:39:50,643 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 20:39:50,643 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 20:39:50,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:38289/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 20:39:50,910 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 20:39:51.367095 24352 mini_ranger_kms.cc:310] Created kms service
I20260806 20:39:51.565788 24352 mini_ranger_kms.cc:326] Created kudu user
I20260806 20:39:51.612682 24352 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 20:39:51 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048788, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:39:52.006254 24352 mini_ranger_kms.cc:384] Added ranger policy
I20260806 20:39:52.007601 24352 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms:/tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 20:39:52.007688 24352 mini_ranger_kms.cc:240] Using host: 127.23.200.20
I20260806 20:39:52.027961 24352 mini_ranger_kms.cc:276] Ranger KMS PID: 3673
I20260806 20:39:52.028175 24352 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.200.20:49631
Aug 06, 2026 8:39:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 8:39:53 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 8:39:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 8:39:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 8:39:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 8:39:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 20:39:54 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786048794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.200.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:54.400372 24352 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.567s user 0.115s sys 0.387s
I20260806 20:39:54.400666 24352 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 20:39:54.400916 24352 mini_ranger_kms.cc:401] 127.23.200.20:49631/kms/v1/keys
Aug 06 20:39:54 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.200.20@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
Aug 06 20:39:54 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048794, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.200.20@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
Aug 06 20:39:57 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:39:57.381563 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:42501
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:42501
--ranger_config_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:49631/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:39:57.828394 3726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:57.828716 3726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:57.828837 3726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:57.837473 3726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:39:57.837612 3726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:57.837693 3726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:57.837776 3726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:39:57.837857 3726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:39:57.849239 3726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.200.20:49631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:42501
--ranger_config_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:42501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:57.850800 3726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:57.852313 3726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:57.866770 3732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:57.867486 3735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:57.867596 3731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:57.868913 3726 server_base.cc:1034] running on GCE node
I20260806 20:39:57.869724 3726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:57.871268 3726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:57.872495 3726 hybrid_clock.cc:648] HybridClock initialized: now 1786048797872452 us; error 32 us; skew 500 ppm
I20260806 20:39:57.872836 3726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:39:57 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:57.880942 3726 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:39:57.882812 3726 webserver.cc:460] Webserver started at http://127.23.200.62:44641/ using document root <none> and password file <none>
I20260806 20:39:57.883598 3726 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:57.883796 3726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:57.884150 3726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:39:57 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:39:57.954931 3726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "1c58a23432284915aaf9084d9e12ba43"
format_stamp: "Formatted at 2026-08-06 20:39:57 on dist-test-slave-tm1g"
server_key: "2fdc49236cf093e538b0c073e1aca728"
server_key_iv: "d3afed15cf5efb6f33e4ef50571223f2"
server_key_version: "kuduclusterkey@0"
I20260806 20:39:57.956106 3726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "1c58a23432284915aaf9084d9e12ba43"
format_stamp: "Formatted at 2026-08-06 20:39:57 on dist-test-slave-tm1g"
server_key: "2fdc49236cf093e538b0c073e1aca728"
server_key_iv: "d3afed15cf5efb6f33e4ef50571223f2"
server_key_version: "kuduclusterkey@0"
I20260806 20:39:57.963670 3726 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.006s
I20260806 20:39:57.980909 3743 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:57.982607 3726 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 20:39:57.982788 3726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "1c58a23432284915aaf9084d9e12ba43"
format_stamp: "Formatted at 2026-08-06 20:39:57 on dist-test-slave-tm1g"
server_key: "2fdc49236cf093e538b0c073e1aca728"
server_key_iv: "d3afed15cf5efb6f33e4ef50571223f2"
server_key_version: "kuduclusterkey@0"
I20260806 20:39:57.983067 3726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:58.008471 3726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:58.013012 3726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:58.013408 3726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:58.042148 3726 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:42501
I20260806 20:39:58.042234 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:42501 every 8 connection(s)
I20260806 20:39:58.043745 3726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:39:58.044225 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 3726
I20260806 20:39:58.044425 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:39:58.045099 24352 external_mini_cluster.cc:1428] Setting key 05f6630946dab9cf129aea59cb868d02
I20260806 20:39:58.051620 3797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:39:58.071578 3797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43: Bootstrap starting.
I20260806 20:39:58.076169 3797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43: Neither blocks nor log segments found. Creating new log.
Aug 06 20:39:58 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:39:58.077585 3797 log.cc:824] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43: Log is configured to *not* fsync() on all Append() calls
I20260806 20:39:58.080979 3797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43: No bootstrap required, opened a new log
I20260806 20:39:58.085700 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:58.047826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47956 (local address 127.23.200.62:42501)
0806 20:39:58.063754 (+ 15928us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:58.063766 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:58.063811 (+ 45us) server_negotiation.cc:407] Connection header received
0806 20:39:58.066074 (+ 2263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:58.066099 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:58.066573 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:58.066904 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:58.068770 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:58.070497 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:58.072341 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:58.073014 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:58.073185 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:58.077848 (+ 4663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:58.077856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:58.077870 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:58.077909 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:58.081281 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:58.081979 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:58.081991 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:58.081997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:58.082189 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:58.082617 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:58.082623 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:58.082634 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:58.083194 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:58.083645 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:58.084132 (+ 487us) server_negotiation.cc:299] Negotiation successful
0806 20:39:58.084559 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15719,"thread_start_us":15227,"threads_started":1}
I20260806 20:39:58.090027 3797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c58a23432284915aaf9084d9e12ba43" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42501 } }
I20260806 20:39:58.090430 3797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:39:58.090547 3797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c58a23432284915aaf9084d9e12ba43, State: Initialized, Role: FOLLOWER
I20260806 20:39:58.091085 3797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c58a23432284915aaf9084d9e12ba43" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42501 } }
I20260806 20:39:58.091421 3797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:39:58.091634 3797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:39:58.091856 3797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:39:58.094200 3797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c58a23432284915aaf9084d9e12ba43" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42501 } }
I20260806 20:39:58.094779 3797 leader_election.cc:302] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1c58a23432284915aaf9084d9e12ba43; no voters:
I20260806 20:39:58.095544 3797 leader_election.cc:288] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:39:58.095641 3802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:39:58.096614 3802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [term 1 LEADER]: Becoming Leader. State: Replica: 1c58a23432284915aaf9084d9e12ba43, State: Running, Role: LEADER
I20260806 20:39:58.097241 3802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c58a23432284915aaf9084d9e12ba43" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42501 } }
I20260806 20:39:58.098119 3797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:39:58.100298 3804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1c58a23432284915aaf9084d9e12ba43. Latest consensus state: current_term: 1 leader_uuid: "1c58a23432284915aaf9084d9e12ba43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c58a23432284915aaf9084d9e12ba43" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42501 } } }
I20260806 20:39:58.100870 3804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:58.100310 3803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1c58a23432284915aaf9084d9e12ba43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1c58a23432284915aaf9084d9e12ba43" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42501 } } }
I20260806 20:39:58.104517 3803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43 [sys.catalog]: This master's current role is: LEADER
I20260806 20:39:58.104724 3808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:39:58.112278 3808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:39:58.113938 3797 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 20:39:58.141759 3808 catalog_manager.cc:1347] Generated new cluster ID: b8e0ad7b47f642e6b9d901ac6a009ca8
I20260806 20:39:58.142067 3808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:39:58.210425 3808 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:39:58.212487 3808 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:39:58.224277 3808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43: Generated new TSK 0
I20260806 20:39:58.225443 3808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:39:58.945938 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:49631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42501
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:39:59 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786048799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 20:39:59.385823 3842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:39:59.386157 3842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:39:59.386358 3842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:39:59.395186 3842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:39:59.395423 3842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:39:59.395650 3842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:39:59.407965 3842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.200.20:49631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:39:59.409593 3842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:39:59.411202 3842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:39:59.438498 3852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:39:59.438934 3853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:59.448210 3842 server_base.cc:1034] running on GCE node
W20260806 20:39:59.448594 3855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:39:59.449635 3842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:39:59.456353 3842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:39:59.457674 3842 hybrid_clock.cc:648] HybridClock initialized: now 1786048799457555 us; error 111 us; skew 500 ppm
I20260806 20:39:59.458091 3842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:39:59 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:39:59.463934 3842 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:39:59.472148 3842 webserver.cc:460] Webserver started at http://127.23.200.1:38989/ using document root <none> and password file <none>
I20260806 20:39:59.473235 3842 fs_manager.cc:359] Metadata directory not provided
I20260806 20:39:59.473392 3842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:39:59.473765 3842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:39:59 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:39:59.506794 3842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "623625ac09154dd2b96b4845422f08af"
format_stamp: "Formatted at 2026-08-06 20:39:59 on dist-test-slave-tm1g"
server_key: "a38c906ed2492e5f9af50c164c5892c6"
server_key_iv: "41f88c8f608ce7fc1faafa59e78921ee"
server_key_version: "kuduclusterkey@0"
I20260806 20:39:59.508203 3842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "623625ac09154dd2b96b4845422f08af"
format_stamp: "Formatted at 2026-08-06 20:39:59 on dist-test-slave-tm1g"
server_key: "a38c906ed2492e5f9af50c164c5892c6"
server_key_iv: "41f88c8f608ce7fc1faafa59e78921ee"
server_key_version: "kuduclusterkey@0"
I20260806 20:39:59.517663 3842 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20260806 20:39:59.557233 3862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:59.559496 3842 fs_manager.cc:725] Time spent opening block manager: real 0.021s user 0.010s sys 0.003s
I20260806 20:39:59.559814 3842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "623625ac09154dd2b96b4845422f08af"
format_stamp: "Formatted at 2026-08-06 20:39:59 on dist-test-slave-tm1g"
server_key: "a38c906ed2492e5f9af50c164c5892c6"
server_key_iv: "41f88c8f608ce7fc1faafa59e78921ee"
server_key_version: "kuduclusterkey@0"
I20260806 20:39:59.560231 3842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:39:59.575232 3842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:39:59.579988 3842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:39:59.580315 3842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:39:59.581480 3842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:39:59.583274 3842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:39:59.583369 3842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:59.583492 3842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:39:59.583566 3842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:39:59.681736 3842 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:34377
I20260806 20:39:59.682029 3978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:34377 every 8 connection(s)
I20260806 20:39:59.684289 3842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:39:59.690166 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 3842
I20260806 20:39:59.690423 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:39:59.691319 24352 external_mini_cluster.cc:1428] Setting key 89a6ba44f8630475b0df263c6672b8ec
Aug 06 20:39:59 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:39:59.735534 3982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:59.690506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:34379 (local address 127.23.200.62:42501)
0806 20:39:59.710761 (+ 20255us) server_negotiation.cc:206] Beginning negotiation
0806 20:39:59.710768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:39:59.714742 (+ 3974us) server_negotiation.cc:407] Connection header received
0806 20:39:59.714810 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:39:59.714816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:39:59.714902 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:39:59.715016 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:39:59.717398 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:59.718612 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:59.721214 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:39:59.722125 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:39:59.722226 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:59.727074 (+ 4848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:39:59.727082 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:39:59.727088 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:39:59.727135 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:39:59.731098 (+ 3963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:59.732157 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:59.732163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:59.732166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:59.732341 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:39:59.732988 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:39:59.732993 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:39:59.732997 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:39:59.733497 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:39:59.733628 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 20:39:59.734881 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0806 20:39:59.735141 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20109,"thread_start_us":19978,"threads_started":1}
I20260806 20:39:59.737919 3981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:39:59.688860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42501 (local address 127.23.200.1:34379)
0806 20:39:59.711530 (+ 22670us) negotiation.cc:107] Waiting for socket to connect
0806 20:39:59.711592 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 20:39:59.714231 (+ 2639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:39:59.715514 (+ 1283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:39:59.715544 (+ 30us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:39:59.716219 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:39:59.717171 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:59.717207 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:59.718833 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:59.718838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:59.721046 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:39:59.721060 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:39:59.722622 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:39:59.722629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:39:59.722933 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:39:59.723798 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:39:59.723830 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:39:59.726780 (+ 2950us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:39:59.731320 (+ 4540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:59.731330 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:59.731348 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:59.731999 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:59.732530 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:39:59.732535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:39:59.732539 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:39:59.732839 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:39:59.733751 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:39:59.733761 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:39:59.734393 (+ 632us) client_negotiation.cc:769] Sending connection context
0806 20:39:59.735883 (+ 1490us) client_negotiation.cc:240] Negotiation successful
0806 20:39:59.736439 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22301,"thread_start_us":34772,"threads_started":1}
I20260806 20:39:59.739763 3979 heartbeater.cc:347] Connected to a master server at 127.23.200.62:42501
I20260806 20:39:59.740154 3979 heartbeater.cc:464] Registering TS with master...
I20260806 20:39:59.741138 3979 heartbeater.cc:511] Master 127.23.200.62:42501 requested a full tablet report, sending...
I20260806 20:39:59.743983 3760 ts_manager.cc:194] Registered new tserver with Master: 623625ac09154dd2b96b4845422f08af (127.23.200.1:34377)
I20260806 20:39:59.746030 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:34379
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:39:59.798118 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:49631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42501
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:39:59 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
Aug 06 20:40:00 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048799, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
W20260806 20:40:00.447750 3985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:00.448297 3985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:00.448508 3985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:00.460239 3985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:00.460459 3985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:00.460717 3985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:40:00.475262 3985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.200.20:49631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:00.477460 3985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:00.479607 3985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:00.502892 4000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:00.510386 3999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:00.512308 4002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:00.512845 3985 server_base.cc:1034] running on GCE node
I20260806 20:40:00.513546 3985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:00.516906 3985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:00.518193 3985 hybrid_clock.cc:648] HybridClock initialized: now 1786048800518101 us; error 51 us; skew 500 ppm
I20260806 20:40:00.518641 3985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:00 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048800, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:00.522343 3985 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:40:00.523979 3985 webserver.cc:460] Webserver started at http://127.23.200.2:39793/ using document root <none> and password file <none>
I20260806 20:40:00.524809 3985 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:00.525000 3985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:00.525348 3985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:40:00 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048800, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:00.542280 3985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "e44776807f6c4dcabf75b8b35b91e9f0"
format_stamp: "Formatted at 2026-08-06 20:40:00 on dist-test-slave-tm1g"
server_key: "f0c5527e0bde467205e8fecdc4618fae"
server_key_iv: "5ae4f5214c6656abaee7e5638487591a"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:00.543339 3985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "e44776807f6c4dcabf75b8b35b91e9f0"
format_stamp: "Formatted at 2026-08-06 20:40:00 on dist-test-slave-tm1g"
server_key: "f0c5527e0bde467205e8fecdc4618fae"
server_key_iv: "5ae4f5214c6656abaee7e5638487591a"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:00.549319 3985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 20:40:00.567950 4010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:00.569871 3985 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260806 20:40:00.570045 3985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "e44776807f6c4dcabf75b8b35b91e9f0"
format_stamp: "Formatted at 2026-08-06 20:40:00 on dist-test-slave-tm1g"
server_key: "f0c5527e0bde467205e8fecdc4618fae"
server_key_iv: "5ae4f5214c6656abaee7e5638487591a"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:00.570276 3985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:00.601724 3985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:00.606786 3985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:00.607190 3985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:00.608568 3985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:00.610324 3985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:00.610443 3985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 20:40:00.610620 3985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:00.610702 3985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:00.674373 3985 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:32963
I20260806 20:40:00.674469 4124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:32963 every 8 connection(s)
I20260806 20:40:00.676510 3985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:40:00.679768 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 3985
I20260806 20:40:00.679944 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:40:00.680629 24352 external_mini_cluster.cc:1428] Setting key daef785421f46c582fc2d4e7ee4ba584
Aug 06 20:40:00 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048800, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:00.710732 4128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:00.682663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:34077 (local address 127.23.200.62:42501)
0806 20:40:00.690695 (+ 8032us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:00.690701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:00.690718 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:40:00.690787 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:00.690792 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:00.690889 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:00.691003 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:00.693238 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:00.694360 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:00.696271 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:00.696848 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:00.697005 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:00.701584 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:00.701590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:00.701593 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:00.701625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:00.706139 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:00.707201 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:00.707209 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:00.707215 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:00.707449 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:00.708048 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:00.708055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:00.708061 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:00.708743 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:00.708902 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:00.709910 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0806 20:40:00.710186 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7907,"thread_start_us":7802,"threads_started":1}
I20260806 20:40:00.711812 4127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:00.681346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42501 (local address 127.23.200.2:34077)
0806 20:40:00.683134 (+ 1788us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:00.683204 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:00.686380 (+ 3176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:00.691338 (+ 4958us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:00.691351 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:00.692255 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:00.693048 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:00.693070 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:00.694540 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:00.694544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:00.696112 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:00.696126 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:00.697182 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:00.697191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:00.697503 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:00.698251 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:00.698275 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:00.701298 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:00.706367 (+ 5069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:00.706377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:00.706397 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:00.707019 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:00.707595 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:00.707600 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:00.707604 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:00.707904 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:00.708890 (+ 986us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:00.708898 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:00.709332 (+ 434us) client_negotiation.cc:769] Sending connection context
0806 20:40:00.709631 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 20:40:00.710199 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":719,"threads_started":1}
I20260806 20:40:00.713466 4125 heartbeater.cc:347] Connected to a master server at 127.23.200.62:42501
I20260806 20:40:00.713815 4125 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:00.714922 4125 heartbeater.cc:511] Master 127.23.200.62:42501 requested a full tablet report, sending...
I20260806 20:40:00.716917 3760 ts_manager.cc:194] Registered new tserver with Master: e44776807f6c4dcabf75b8b35b91e9f0 (127.23.200.2:32963)
I20260806 20:40:00.718199 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:34077
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
I20260806 20:40:00.749725 3979 heartbeater.cc:503] Master 127.23.200.62:42501 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:40:00.753515 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:49631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42501
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:40:01.305449 4131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:01.305768 4131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:01.305975 4131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:01.315269 4131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:01.315480 4131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:01.315773 4131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:40:01.328590 4131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.200.20:49631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42501
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:01.330237 4131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:01.331794 4131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:01.349006 4136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:01.349272 4140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:01.349313 4137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:01.351096 4131 server_base.cc:1034] running on GCE node
I20260806 20:40:01.351676 4131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:01.352794 4131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:01.354050 4131 hybrid_clock.cc:648] HybridClock initialized: now 1786048801353962 us; error 43 us; skew 500 ppm
I20260806 20:40:01.354503 4131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:01 dist-test-slave-tm1g krb5kdc[3257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048801, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:01.358780 4131 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:40:01.360575 4131 webserver.cc:460] Webserver started at http://127.23.200.3:40327/ using document root <none> and password file <none>
I20260806 20:40:01.361366 4131 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:01.361478 4131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:01.361804 4131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:40:01 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048801, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:01.377085 4131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "12bb97d263584653a5705b9e0e93dc78"
format_stamp: "Formatted at 2026-08-06 20:40:01 on dist-test-slave-tm1g"
server_key: "403150ecf15bd0bd4eb4d82f6496cdc9"
server_key_iv: "86a676a910c0074417ce1a96405d2e99"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:01.378089 4131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "12bb97d263584653a5705b9e0e93dc78"
format_stamp: "Formatted at 2026-08-06 20:40:01 on dist-test-slave-tm1g"
server_key: "403150ecf15bd0bd4eb4d82f6496cdc9"
server_key_iv: "86a676a910c0074417ce1a96405d2e99"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:01.385608 4131 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.001s
I20260806 20:40:01.403357 4147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:01.404874 4131 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 20:40:01.405119 4131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "12bb97d263584653a5705b9e0e93dc78"
format_stamp: "Formatted at 2026-08-06 20:40:01 on dist-test-slave-tm1g"
server_key: "403150ecf15bd0bd4eb4d82f6496cdc9"
server_key_iv: "86a676a910c0074417ce1a96405d2e99"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:01.405395 4131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:01.420406 4131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:01.425750 4131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:01.426113 4131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:01.427448 4131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:01.429913 4131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:01.430003 4131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:01.430172 4131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:01.430253 4131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:01.508737 4131 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37863
I20260806 20:40:01.508780 4261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37863 every 8 connection(s)
I20260806 20:40:01.510471 4131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:40:01.513939 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 4131
I20260806 20:40:01.514122 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:40:01.514743 24352 external_mini_cluster.cc:1428] Setting key 6a1b7ac6db71fa97649ef2054ebce7e3
Aug 06 20:40:01 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048801, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:01.531750 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.513141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:55317 (local address 127.23.200.62:42501)
0806 20:40:01.513793 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.513799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.515033 (+ 1234us) server_negotiation.cc:407] Connection header received
0806 20:40:01.517250 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.517255 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.517359 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.517482 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:01.519130 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.519977 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.521608 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.522056 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.522153 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.525753 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:01.525759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:01.525762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:01.525802 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:01.528645 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:01.529506 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:01.529509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:01.529512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:01.529676 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:01.530204 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:01.530209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:01.530212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:01.530607 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:01.530717 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.531311 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.531481 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":373,"threads_started":1}
I20260806 20:40:01.532693 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.513831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42501 (local address 127.23.200.3:55317)
0806 20:40:01.514857 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:01.514903 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:01.516985 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:01.517676 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:01.517684 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:01.518260 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:01.518982 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.519001 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.520105 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.520109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.521465 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.521477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.522280 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.522288 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.522634 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.523333 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:01.523355 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:01.525547 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:01.528793 (+ 3246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:01.528812 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:01.528830 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:01.529386 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:01.529817 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:01.529824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:01.529827 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:01.530085 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:01.530725 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:01.530730 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:01.531013 (+ 283us) client_negotiation.cc:769] Sending connection context
0806 20:40:01.531223 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 20:40:01.531630 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":439,"threads_started":1}
I20260806 20:40:01.534091 4262 heartbeater.cc:347] Connected to a master server at 127.23.200.62:42501
I20260806 20:40:01.534458 4262 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:01.535315 4262 heartbeater.cc:511] Master 127.23.200.62:42501 requested a full tablet report, sending...
I20260806 20:40:01.536641 3760 ts_manager.cc:194] Registered new tserver with Master: 12bb97d263584653a5705b9e0e93dc78 (127.23.200.3:37863)
I20260806 20:40:01.537492 3760 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:55317
I20260806 20:40:01.542596 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:40:01.560451 4264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.547273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.23.200.62:42501)
0806 20:40:01.547521 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.547528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.547836 (+ 308us) server_negotiation.cc:407] Connection header received
0806 20:40:01.548044 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.548052 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.548114 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.548212 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:01.550017 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.550804 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.552225 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.552664 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.552801 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.555313 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:01.555318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:01.555321 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:01.555359 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:01.557807 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:01.558494 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:01.558501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:01.558504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:01.558636 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:01.559046 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:01.559052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:01.559056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:01.559313 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:01.559579 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.559886 (+ 307us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.560108 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 20:40:01.564404 3760 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:40:01.572772 3760 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:40:01.616814 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.587745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.23.200.1:34377)
0806 20:40:01.606122 (+ 18377us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.606135 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.607377 (+ 1242us) server_negotiation.cc:407] Connection header received
0806 20:40:01.607578 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.607593 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.607975 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.608278 (+ 303us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:01.609445 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.611456 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.614225 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.615441 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.615620 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.615856 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.616237 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.616427 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18122,"thread_start_us":17929,"threads_started":1}
I20260806 20:40:01.617058 4277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.586946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:34377 (local address 127.0.0.1:48244)
0806 20:40:01.606683 (+ 19737us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:01.606709 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:01.606890 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:01.608657 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:01.608667 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:01.608725 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:01.609246 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.609260 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.611666 (+ 2406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.611671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.614028 (+ 2357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.614041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.615740 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.615747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.615965 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.616022 (+ 57us) client_negotiation.cc:769] Sending connection context
0806 20:40:01.616192 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 20:40:01.616382 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19556,"mutex_wait_us":2,"thread_start_us":20055,"threads_started":1}
I20260806 20:40:01.618861 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.587658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59352 (local address 127.23.200.2:32963)
0806 20:40:01.608152 (+ 20494us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.608160 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.608890 (+ 730us) server_negotiation.cc:407] Connection header received
0806 20:40:01.609136 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.609144 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.609450 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.609703 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:01.611269 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.612478 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.615381 (+ 2903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.616921 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.617056 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.617218 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.617859 (+ 641us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.618086 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20311,"thread_start_us":20160,"threads_started":1}
I20260806 20:40:01.618917 4280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.587676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32963 (local address 127.0.0.1:59352)
0806 20:40:01.608723 (+ 21047us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:01.608742 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:01.608922 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:01.609916 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:01.609920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:01.609976 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:01.611038 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.611054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.612669 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.612675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.615194 (+ 2519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.615210 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.617386 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.617393 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.617624 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.617662 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:40:01.617752 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 20:40:01.618502 (+ 750us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20907,"mutex_wait_us":119,"thread_start_us":20072,"threads_started":1}
I20260806 20:40:01.620128 4276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.586470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37863 (local address 127.0.0.1:36580)
0806 20:40:01.603222 (+ 16752us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:01.603270 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:01.603527 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:01.609678 (+ 6151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:01.609686 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:01.609715 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:01.610418 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.610433 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.613136 (+ 2703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.613142 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.615452 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.615463 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.618058 (+ 2595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.618066 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.619215 (+ 1149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.619248 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 20:40:01.619350 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 20:40:01.619497 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16381,"mutex_wait_us":2,"spinlock_wait_cycles":41728,"thread_start_us":16117,"threads_started":1}
I20260806 20:40:01.621035 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.586950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.23.200.3:37863)
0806 20:40:01.608931 (+ 21981us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.608938 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.608999 (+ 61us) server_negotiation.cc:407] Connection header received
0806 20:40:01.609101 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.609111 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.609407 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.610530 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:01.611308 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.612989 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.615601 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.617685 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.618899 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.619123 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.620373 (+ 1250us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.620532 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21726,"thread_start_us":21427,"threads_started":1}
I20260806 20:40:01.621837 3910 tablet_service.cc:1467] Processing CreateTablet for tablet d71ad3221ecc4e81a6635732035d9a01 (DEFAULT_TABLE table=test-table [id=1d7849fef2d343248b7fb7e29a6ab26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:40:01.623258 4059 tablet_service.cc:1467] Processing CreateTablet for tablet d71ad3221ecc4e81a6635732035d9a01 (DEFAULT_TABLE table=test-table [id=1d7849fef2d343248b7fb7e29a6ab26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:40:01.623965 3910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71ad3221ecc4e81a6635732035d9a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:01.624924 4059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71ad3221ecc4e81a6635732035d9a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:01.624841 4195 tablet_service.cc:1467] Processing CreateTablet for tablet d71ad3221ecc4e81a6635732035d9a01 (DEFAULT_TABLE table=test-table [id=1d7849fef2d343248b7fb7e29a6ab26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:40:01.626142 4195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d71ad3221ecc4e81a6635732035d9a01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:01.639842 4282 tablet_bootstrap.cc:492] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Bootstrap starting.
I20260806 20:40:01.642676 4283 tablet_bootstrap.cc:492] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Bootstrap starting.
I20260806 20:40:01.644891 4284 tablet_bootstrap.cc:492] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Bootstrap starting.
I20260806 20:40:01.646198 4282 tablet_bootstrap.cc:654] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:01.647682 4283 tablet_bootstrap.cc:654] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:01.647897 4282 log.cc:824] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:01.648427 4284 tablet_bootstrap.cc:654] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:01.649909 4283 log.cc:824] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:01.650806 4284 log.cc:824] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:01.652784 4282 tablet_bootstrap.cc:492] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: No bootstrap required, opened a new log
I20260806 20:40:01.653069 4282 ts_tablet_manager.cc:1397] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 20:40:01.654737 4284 tablet_bootstrap.cc:492] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: No bootstrap required, opened a new log
I20260806 20:40:01.655063 4284 ts_tablet_manager.cc:1397] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260806 20:40:01.655577 4283 tablet_bootstrap.cc:492] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: No bootstrap required, opened a new log
I20260806 20:40:01.656038 4283 ts_tablet_manager.cc:1397] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260806 20:40:01.665144 4284 raft_consensus.cc:348] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.665766 4284 raft_consensus.cc:374] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:01.665956 4284 raft_consensus.cc:729] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 623625ac09154dd2b96b4845422f08af, State: Initialized, Role: FOLLOWER
I20260806 20:40:01.666867 4284 consensus_queue.cc:260] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.668516 4282 raft_consensus.cc:348] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.669271 4282 raft_consensus.cc:374] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:01.669499 4282 raft_consensus.cc:729] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12bb97d263584653a5705b9e0e93dc78, State: Initialized, Role: FOLLOWER
I20260806 20:40:01.669538 4284 ts_tablet_manager.cc:1428] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260806 20:40:01.670676 4282 consensus_queue.cc:260] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.671880 4283 raft_consensus.cc:348] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.672559 4283 raft_consensus.cc:374] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:01.672840 4283 raft_consensus.cc:729] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e44776807f6c4dcabf75b8b35b91e9f0, State: Initialized, Role: FOLLOWER
I20260806 20:40:01.672919 4262 heartbeater.cc:503] Master 127.23.200.62:42501 was elected leader, sending a full tablet report...
I20260806 20:40:01.675253 4282 ts_tablet_manager.cc:1428] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Time spent starting tablet: real 0.022s user 0.012s sys 0.008s
I20260806 20:40:01.673694 4283 consensus_queue.cc:260] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.679297 4125 heartbeater.cc:503] Master 127.23.200.62:42501 was elected leader, sending a full tablet report...
I20260806 20:40:01.680166 4283 ts_tablet_manager.cc:1428] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Time spent starting tablet: real 0.024s user 0.015s sys 0.005s
W20260806 20:40:01.680209 4126 tablet.cc:2369] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:40:01.711066 3980 tablet.cc:2369] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:40:01.762776 4263 tablet.cc:2369] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:40:01.880702 4288 raft_consensus.cc:482] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:40:01.881174 4288 raft_consensus.cc:504] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.883520 4288 leader_election.cc:288] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12bb97d263584653a5705b9e0e93dc78 (127.23.200.3:37863), e44776807f6c4dcabf75b8b35b91e9f0 (127.23.200.2:32963)
I20260806 20:40:01.893754 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.884062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38977 (local address 127.23.200.2:32963)
0806 20:40:01.884315 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.884320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.885721 (+ 1401us) server_negotiation.cc:407] Connection header received
0806 20:40:01.886001 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.886008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.886129 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.886240 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:01.887400 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.888725 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.891096 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.892575 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.892734 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.892788 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.893409 (+ 621us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.893505 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:40:01.894109 4291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.884121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32963 (local address 127.23.200.1:38977)
0806 20:40:01.885573 (+ 1452us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:01.885604 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:01.885799 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:01.886270 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:01.886274 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:01.886339 (+ 65us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:01.887200 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.887213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.888866 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.888872 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.890881 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.890897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.892941 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.892949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.893181 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.893216 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:40:01.893356 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 20:40:01.893525 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"thread_start_us":781,"threads_started":1}
I20260806 20:40:01.895498 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.883678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:37499 (local address 127.23.200.3:37863)
0806 20:40:01.883978 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.883986 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.886210 (+ 2224us) server_negotiation.cc:407] Connection header received
0806 20:40:01.886425 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.886432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.886516 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.886662 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:01.888168 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.889426 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.892754 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.894617 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.894879 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.894956 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.895171 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.895265 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 20:40:01.895969 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.884477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37863 (local address 127.23.200.1:37499)
0806 20:40:01.886055 (+ 1578us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:01.886095 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:01.886267 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:01.886661 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:01.886664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:01.886685 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:01.887972 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.887997 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.889561 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.889564 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.892519 (+ 2955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:01.892535 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.894859 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.894864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:01.894983 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.895039 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 20:40:01.895148 (+ 109us) client_negotiation.cc:240] Negotiation successful
0806 20:40:01.895286 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"mutex_wait_us":4,"thread_start_us":1119,"threads_started":1}
I20260806 20:40:01.895841 4080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71ad3221ecc4e81a6635732035d9a01" candidate_uuid: "623625ac09154dd2b96b4845422f08af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" is_pre_election: true
I20260806 20:40:01.896669 4080 raft_consensus.cc:2454] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623625ac09154dd2b96b4845422f08af in term 0.
I20260806 20:40:01.897090 4217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71ad3221ecc4e81a6635732035d9a01" candidate_uuid: "623625ac09154dd2b96b4845422f08af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bb97d263584653a5705b9e0e93dc78" is_pre_election: true
I20260806 20:40:01.897666 3863 leader_election.cc:302] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623625ac09154dd2b96b4845422f08af, e44776807f6c4dcabf75b8b35b91e9f0; no voters:
I20260806 20:40:01.897836 4217 raft_consensus.cc:2454] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 623625ac09154dd2b96b4845422f08af in term 0.
I20260806 20:40:01.898273 4288 raft_consensus.cc:2781] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:40:01.898568 4288 raft_consensus.cc:482] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:40:01.898766 4288 raft_consensus.cc:3037] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:01.902479 4288 raft_consensus.cc:504] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.903257 4288 leader_election.cc:288] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [CANDIDATE]: Term 1 election: Requested vote from peers 12bb97d263584653a5705b9e0e93dc78 (127.23.200.3:37863), e44776807f6c4dcabf75b8b35b91e9f0 (127.23.200.2:32963)
I20260806 20:40:01.903690 4080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71ad3221ecc4e81a6635732035d9a01" candidate_uuid: "623625ac09154dd2b96b4845422f08af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e44776807f6c4dcabf75b8b35b91e9f0"
I20260806 20:40:01.903774 4217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d71ad3221ecc4e81a6635732035d9a01" candidate_uuid: "623625ac09154dd2b96b4845422f08af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12bb97d263584653a5705b9e0e93dc78"
I20260806 20:40:01.903853 4080 raft_consensus.cc:3037] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:01.904049 4217 raft_consensus.cc:3037] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:01.906841 4080 raft_consensus.cc:2454] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623625ac09154dd2b96b4845422f08af in term 1.
I20260806 20:40:01.906955 4217 raft_consensus.cc:2454] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 623625ac09154dd2b96b4845422f08af in term 1.
I20260806 20:40:01.907394 3863 leader_election.cc:302] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 623625ac09154dd2b96b4845422f08af, e44776807f6c4dcabf75b8b35b91e9f0; no voters:
I20260806 20:40:01.907774 4288 raft_consensus.cc:2781] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:40:01.908663 4288 raft_consensus.cc:686] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 1 LEADER]: Becoming Leader. State: Replica: 623625ac09154dd2b96b4845422f08af, State: Running, Role: LEADER
I20260806 20:40:01.909211 4288 consensus_queue.cc:237] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } }
I20260806 20:40:01.914942 3759 catalog_manager.cc:5495] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af reported cstate change: term changed from 0 to 1, leader changed from <none> to 623625ac09154dd2b96b4845422f08af (127.23.200.1). New cstate: current_term: 1 leader_uuid: "623625ac09154dd2b96b4845422f08af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "623625ac09154dd2b96b4845422f08af" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 } health_report { overall_health: UNKNOWN } } }
I20260806 20:40:01.987062 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:01.980390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48246 (local address 127.23.200.1:34377)
0806 20:40:01.980735 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:01.980741 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:01.980762 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:40:01.980923 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:01.980928 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:01.981042 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:01.981202 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:40:01.981667 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.982806 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.984460 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:01.985081 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:01.985223 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:01.985852 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:40:01.986481 (+ 629us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:40:01.986642 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:01.986709 (+ 67us) server_negotiation.cc:299] Negotiation successful
0806 20:40:01.986788 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:40:01.997018 4080 raft_consensus.cc:1261] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 1 FOLLOWER]: Refusing update from remote peer 623625ac09154dd2b96b4845422f08af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:40:01.996981 4217 raft_consensus.cc:1261] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 1 FOLLOWER]: Refusing update from remote peer 623625ac09154dd2b96b4845422f08af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:40:01.998189 4293 consensus_queue.cc:1035] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [LEADER]: Connected to new peer: Peer: permanent_uuid: "12bb97d263584653a5705b9e0e93dc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:40:01.998735 4288 consensus_queue.cc:1035] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e44776807f6c4dcabf75b8b35b91e9f0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:40:02.013175 4297 mvcc.cc:205] Tried to move back new op lower bound from 7315655892967428096 to 7315655892624785408. Current Snapshot: MvccSnapshot[applied={T|T < 7315655892967428096}]
I20260806 20:40:02.035288 3760 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:40:02.035641 3760 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:40:02.039695 3760 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1c58a23432284915aaf9084d9e12ba43: Sending DeleteTablet for 3 replicas of tablet d71ad3221ecc4e81a6635732035d9a01
I20260806 20:40:02.041051 4195 tablet_service.cc:1514] Processing DeleteTablet for tablet d71ad3221ecc4e81a6635732035d9a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:40:02 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:36580
I20260806 20:40:02.042372 3910 tablet_service.cc:1514] Processing DeleteTablet for tablet d71ad3221ecc4e81a6635732035d9a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:40:02 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:48244
I20260806 20:40:02.045270 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 3842
I20260806 20:40:02.046717 4059 tablet_service.cc:1514] Processing DeleteTablet for tablet d71ad3221ecc4e81a6635732035d9a01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:40:02 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:59352
I20260806 20:40:02.053598 4307 tablet_replica.cc:330] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: stopping tablet replica
I20260806 20:40:02.054134 4307 raft_consensus.cc:2229] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:40:02.054800 4307 raft_consensus.cc:2258] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:40:02.057173 4307 ts_tablet_manager.cc:1905] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:40:02.061400 4310 tablet_replica.cc:330] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: stopping tablet replica
I20260806 20:40:02.061777 4310 raft_consensus.cc:2229] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:40:02.062204 4310 raft_consensus.cc:2258] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:40:02.063076 4308 tablet_replica.cc:330] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: stopping tablet replica
I20260806 20:40:02.063515 4308 raft_consensus.cc:2229] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:40:02.064371 4308 raft_consensus.cc:2258] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:40:02.066457 4310 ts_tablet_manager.cc:1905] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:40:02.066461 4308 ts_tablet_manager.cc:1905] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:40:02.070554 4307 ts_tablet_manager.cc:1918] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:40:02.070843 4307 log.cc:1196] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/d71ad3221ecc4e81a6635732035d9a01
I20260806 20:40:02.071460 4307 ts_tablet_manager.cc:1939] T d71ad3221ecc4e81a6635732035d9a01 P 12bb97d263584653a5705b9e0e93dc78: Deleting consensus metadata
I20260806 20:40:02.073858 3747 catalog_manager.cc:4841] TS 12bb97d263584653a5705b9e0e93dc78 (127.23.200.3:37863): tablet d71ad3221ecc4e81a6635732035d9a01 (table test-table [id=1d7849fef2d343248b7fb7e29a6ab26f]) successfully deleted
Aug 06 20:40:02 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:40:02.079927 4310 ts_tablet_manager.cc:1918] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:40:02.080299 4310 log.cc:1196] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/d71ad3221ecc4e81a6635732035d9a01
I20260806 20:40:02.081102 4310 ts_tablet_manager.cc:1939] T d71ad3221ecc4e81a6635732035d9a01 P e44776807f6c4dcabf75b8b35b91e9f0: Deleting consensus metadata
I20260806 20:40:02.083504 3744 catalog_manager.cc:4841] TS e44776807f6c4dcabf75b8b35b91e9f0 (127.23.200.2:32963): tablet d71ad3221ecc4e81a6635732035d9a01 (table test-table [id=1d7849fef2d343248b7fb7e29a6ab26f]) successfully deleted
I20260806 20:40:02.088837 4281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:02.046722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48260 (local address 127.23.200.1:34377)
0806 20:40:02.047002 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:02.047011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:02.063718 (+ 16707us) server_negotiation.cc:407] Connection header received
0806 20:40:02.064200 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:02.064206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:02.064288 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:02.064380 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:02.067054 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:02.068778 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:02.071216 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:02.072340 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:02.072497 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:02.079282 (+ 6785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:02.079297 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:02.079326 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:02.079392 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:02.084226 (+ 4834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:02.085253 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:02.085261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:02.085265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:02.085708 (+ 443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:02.086320 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:02.086325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:02.086328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:02.086934 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:02.087409 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:02.088172 (+ 763us) server_negotiation.cc:299] Negotiation successful
0806 20:40:02.088467 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:40:02.089666 4308 ts_tablet_manager.cc:1918] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:40:02.089936 4308 log.cc:1196] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/d71ad3221ecc4e81a6635732035d9a01
I20260806 20:40:02.090827 3974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:02.090858 4308 ts_tablet_manager.cc:1939] T d71ad3221ecc4e81a6635732035d9a01 P 623625ac09154dd2b96b4845422f08af: Deleting consensus metadata
I20260806 20:40:02.192591 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 3842
I20260806 20:40:02.192749 3745 catalog_manager.cc:4841] TS 623625ac09154dd2b96b4845422f08af (127.23.200.1:34377): tablet d71ad3221ecc4e81a6635732035d9a01 (table test-table [id=1d7849fef2d343248b7fb7e29a6ab26f]) successfully deleted
I20260806 20:40:02.211656 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 3985
Aug 06 20:40:02 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:40:02.226706 4279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:02.212438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59366 (local address 127.23.200.2:32963)
0806 20:40:02.212610 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:02.212615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:02.212686 (+ 71us) server_negotiation.cc:407] Connection header received
0806 20:40:02.212841 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:02.212845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:02.212915 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:02.213001 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:02.214256 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:02.215026 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:02.216289 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:02.216688 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:02.216794 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:02.221147 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:02.221159 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:02.221174 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:02.221218 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:02.223995 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:02.224702 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:02.224716 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:02.224719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:02.224877 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:02.225193 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:02.225197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:02.225199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:02.225610 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:02.225837 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:02.226034 (+ 197us) server_negotiation.cc:299] Negotiation successful
0806 20:40:02.226202 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 20:40:02.227327 4120 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:02.317262 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 3985
I20260806 20:40:02.335192 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 4131
Aug 06 20:40:02 dist-test-slave-tm1g krb5kdc[3257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048797, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:40:02.350889 4278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:02.336021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36590 (local address 127.23.200.3:37863)
0806 20:40:02.336223 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:02.336230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:02.336291 (+ 61us) server_negotiation.cc:407] Connection header received
0806 20:40:02.336515 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:02.336520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:02.336598 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:02.336708 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:02.338027 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:02.338820 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:02.340108 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:02.340506 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:02.340638 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:02.345128 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:02.345143 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:02.345160 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:02.345193 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:02.348090 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:02.348773 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:02.348780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:02.348783 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:02.348917 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:02.349294 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:02.349299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:02.349302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:02.349657 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:02.349960 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:02.350266 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 20:40:02.350525 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 20:40:02.351545 4257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:02.440232 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 4131
I20260806 20:40:02.457931 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 3726
I20260806 20:40:02.459223 3792 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:02.571735 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 3726
I20260806 20:40:02.598697 24352 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 20:40:02.943418 24352 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 20:40:02.943710 24352 mini_ranger.cc:66] Stopping Ranger...
I20260806 20:40:03.288795 24352 mini_ranger.cc:68] Stopped Ranger
2026-08-06 20:40:03.289 UTC [3288] LOG: received smart shutdown request
2026-08-06 20:40:03.292 UTC [3288] LOG: background worker "logical replication launcher" (PID 3298) exited with exit code 1
2026-08-06 20:40:03.293 UTC [3292] LOG: shutting down
2026-08-06 20:40:03.450 UTC [3288] LOG: database system is shut down
2026-08-06T20:40:03Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51199 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:40:03 dist-test-slave-tm1g krb5kdc[4323](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:40:03 dist-test-slave-tm1g krb5kdc[4323](info): set up 2 sockets
Aug 06 20:40:03 dist-test-slave-tm1g krb5kdc[4323](info): commencing operation
krb5kdc: starting...
W20260806 20:40:04.717994 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:40:04 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048804, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:40:04Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/rangeradmin_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/rangeradmin_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/rangerlookup_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/rangerlookup_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
I20260806 20:40:05.035161 24352 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskG_AEDx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/postgres -l logfile start
2026-08-06 20:40:07.355 UTC [4354] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 20:40:07.355 UTC [4354] LOG: listening on IPv4 address "127.23.200.20", port 40561
2026-08-06 20:40:07.358 UTC [4354] LOG: listening on Unix socket "/tmp/.s.PGSQL.40561"
2026-08-06 20:40:07.380 UTC [4357] LOG: database system was shut down at 2026-08-06 20:40:06 UTC
2026-08-06 20:40:07.383 UTC [4354] LOG: database system is ready to accept connections
I20260806 20:40:08.369657 24352 mini_postgres.cc:96] Postgres bound to 40561
2026-08-06 20:40:08.388 UTC [4365] FATAL: database "slave" does not exist
127.23.200.20:40561 - accepting connections
I20260806 20:40:08.389533 24352 mini_ranger.cc:161] Starting Ranger...
I20260806 20:40:08.417232 24352 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 20:40:08.857112 24352 mini_ranger.cc:87] Created ranger Postgres database
I20260806 20:40:08.857234 24352 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-admin
2026-08-06 20:40:09,164 [I] DB FLAVOR :POSTGRES
2026-08-06 20:40:09,164 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 20:40:09,164 [I] Checking connection..
2026-08-06 20:40:09,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 20:40:09,408 [I] Checking connection passed.
2026-08-06 20:40:09,408 [I] --------- Verifying version history table ---------
2026-08-06 20:40:09,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 20:40:09,661 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 20:40:09,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 20:40:09,902 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 20:40:09,902 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 20:40:09,902 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 20:40:10.150 UTC [4431] WARNING: there is no transaction in progress
2026-08-06 20:40:10,156 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 20:40:10,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 20:40:10,410 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 20:40:10,411 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 20:40:10,411 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 20:40:10,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 20:40:10,934 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tm1g.c.gcp-upstream.internal','N') ;"
2026-08-06 20:40:11,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 20:40:11,403 [I] Table x_portal_user does not exist in database ranger
2026-08-06 20:40:11,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 20:40:11,663 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 20:40:11,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 20:40:11,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 20:40:12,165 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 20:40:12,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 20:40:13.064 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.079 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.094 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.107 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.121 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.137 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.198 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.204 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.217 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.227 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.237 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.249 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.258 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.263 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.269 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13.275 UTC [4543] WARNING: there is no transaction in progress
2026-08-06 20:40:13,835 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 20:40:13,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 20:40:14,072 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 20:40:14,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 20:40:14,328 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 20:40:14,328 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 20:40:14,580 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 20:40:14,581 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 20:40:14,825 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 20:40:15,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tm1g.c.gcp-upstream.internal';"
2026-08-06 20:40:15,311 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 20:40:15,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 20:40:15,572 [I] DB_PATCHES have already been applied
I20260806 20:40:15.578128 24352 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-admin:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskG_AEDx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 20:40:15.578220 24352 mini_ranger.cc:193] Using host: 127.23.200.20
I20260806 20:40:15.594745 24352 mini_ranger.cc:227] Ranger admin URL: http://127.23.200.20:44227
Aug 06, 2026 8:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 8:40:15 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 8:40:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 8:40:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 8:40:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 8:40:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 8:40:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 20:40:16.748423 24352 mini_ranger.cc:160] Time spent starting Ranger: real 8.359s user 0.007s sys 0.036s
Aug 06, 2026 8:40:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 8:40:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 06, 2026 8:40:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 8:40:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 8:40:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 20:40:37.413793 24352 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.200.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.200.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/rangerkms_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.200.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/rangerkms_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.200.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.200.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.200.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/HTTP_127.23.200.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 20:40:37 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048837, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 20:40:37.596581 24352 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 20:40:37.618947 24352 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 20:40:38.492745 24352 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 20:40:38.492868 24352 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms
I20260806 20:40:38.492913 24352 mini_ranger_kms.cc:227] Using postgres at 127.23.200.20:40561
2026-08-06 20:40:38,884 [I] DB FLAVOR :POSTGRES
2026-08-06 20:40:38,884 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 20:40:38,884 [I] Checking connection
2026-08-06 20:40:38,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 20:40:39,115 [I] connection success
2026-08-06 20:40:39,116 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 20:40:39,116 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 20:40:39,116 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 20:40:39,344 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 20:40:39,345 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 20:40:39,345 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 20:40:39,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.200.20:40561/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 20:40:39,609 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 20:40:40.073704 24352 mini_ranger_kms.cc:310] Created kms service
I20260806 20:40:40.254002 24352 mini_ranger_kms.cc:326] Created kudu user
I20260806 20:40:40.299448 24352 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 20:40:40 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048837, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:40.703089 24352 mini_ranger_kms.cc:384] Added ranger policy
I20260806 20:40:40.703441 24352 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms:/tmp/dist-test-taskG_AEDx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskG_AEDx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 20:40:40.703498 24352 mini_ranger_kms.cc:240] Using host: 127.23.200.20
I20260806 20:40:40.724076 24352 mini_ranger_kms.cc:276] Ranger KMS PID: 4727
I20260806 20:40:40.724192 24352 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.200.20:56417
Aug 06, 2026 8:40:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 8:40:41 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 8:40:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 8:40:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 8:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 8:40:42 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 20:40:42 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786048842, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.200.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:43.071393 24352 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.475s user 0.101s sys 0.313s
I20260806 20:40:43.071657 24352 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 20:40:43.071887 24352 mini_ranger_kms.cc:401] 127.23.200.20:56417/kms/v1/keys
Aug 06 20:40:43 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048842, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.200.20@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
Aug 06 20:40:43 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048842, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.200.20@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
Aug 06 20:40:45 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:40:46.001453 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:43657
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:43657
--ranger_config_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:40:46.510787 4779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:46.511294 4779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:46.511524 4779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:46.524585 4779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:40:46.524916 4779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:46.525085 4779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:46.525260 4779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:40:46.525370 4779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:40:46.544826 4779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.200.20:56417/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:43657
--ranger_config_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:46.547348 4779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:46.549857 4779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:46.577847 4787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:46.578080 4785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:46.579587 4784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:46.584110 4779 server_base.cc:1034] running on GCE node
I20260806 20:40:46.586369 4779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:46.590131 4779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:46.591490 4779 hybrid_clock.cc:648] HybridClock initialized: now 1786048846591399 us; error 62 us; skew 500 ppm
I20260806 20:40:46.592253 4779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:46 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:46.602152 4779 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:40:46.605043 4779 webserver.cc:460] Webserver started at http://127.23.200.62:46535/ using document root <none> and password file <none>
I20260806 20:40:46.606240 4779 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:46.606513 4779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:46.607057 4779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:40:46 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048846, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:46.695782 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "436a4f1fe35f4a6083332451e97a7626"
format_stamp: "Formatted at 2026-08-06 20:40:46 on dist-test-slave-tm1g"
server_key: "8a9f1fece5fac304c82da85f0147e026"
server_key_iv: "dd2ba6b3c3efac7462213dd5bc167b31"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:46.698182 4779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "436a4f1fe35f4a6083332451e97a7626"
format_stamp: "Formatted at 2026-08-06 20:40:46 on dist-test-slave-tm1g"
server_key: "8a9f1fece5fac304c82da85f0147e026"
server_key_iv: "dd2ba6b3c3efac7462213dd5bc167b31"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:46.712881 4779 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.010s sys 0.001s
I20260806 20:40:46.744412 4794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:46.747114 4779 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260806 20:40:46.747442 4779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "436a4f1fe35f4a6083332451e97a7626"
format_stamp: "Formatted at 2026-08-06 20:40:46 on dist-test-slave-tm1g"
server_key: "8a9f1fece5fac304c82da85f0147e026"
server_key_iv: "dd2ba6b3c3efac7462213dd5bc167b31"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:46.747838 4779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:46.803575 4779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:46.810230 4779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:46.810819 4779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:46.844548 4779 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:43657
I20260806 20:40:46.844755 4848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:43657 every 8 connection(s)
I20260806 20:40:46.848026 4779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:40:46.854216 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 4779
I20260806 20:40:46.854669 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:40:46.856568 24352 external_mini_cluster.cc:1428] Setting key a0b535c6cfd0e92ee20782752b6dca0c
I20260806 20:40:46.856778 4849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 20:40:46 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:46.887988 4849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Bootstrap starting.
I20260806 20:40:46.894089 4849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:46.896075 4849 log.cc:824] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:46.903187 4849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: No bootstrap required, opened a new log
I20260806 20:40:46.904928 4852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:46.860160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58676 (local address 127.23.200.62:43657)
0806 20:40:46.861644 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:46.861667 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:46.861749 (+ 82us) server_negotiation.cc:407] Connection header received
0806 20:40:46.865440 (+ 3691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:46.865482 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:46.866221 (+ 739us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:46.866917 (+ 696us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:46.870814 (+ 3897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:46.874471 (+ 3657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:46.878448 (+ 3977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:46.879808 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:46.880070 (+ 262us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:46.889728 (+ 9658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:46.889747 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:46.889772 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:46.889840 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:46.895403 (+ 5563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:46.896798 (+ 1395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:46.896806 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:46.896822 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:46.897077 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:46.897823 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:46.897834 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:46.897836 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:46.899286 (+ 1450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:46.899901 (+ 615us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:46.900754 (+ 853us) server_negotiation.cc:299] Negotiation successful
0806 20:40:46.901603 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1093,"thread_start_us":706,"threads_started":1}
I20260806 20:40:46.918630 4849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:46.919220 4849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:46.919332 4849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436a4f1fe35f4a6083332451e97a7626, State: Initialized, Role: FOLLOWER
I20260806 20:40:46.920282 4849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:46.920630 4849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:40:46.920776 4849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:40:46.921002 4849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:46.924928 4849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:46.925933 4849 leader_election.cc:302] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 436a4f1fe35f4a6083332451e97a7626; no voters:
I20260806 20:40:46.927054 4849 leader_election.cc:288] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:40:46.927292 4854 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:40:46.928547 4854 raft_consensus.cc:686] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 LEADER]: Becoming Leader. State: Replica: 436a4f1fe35f4a6083332451e97a7626, State: Running, Role: LEADER
I20260806 20:40:46.929442 4854 consensus_queue.cc:237] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:46.931011 4849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:40:46.936671 4856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436a4f1fe35f4a6083332451e97a7626. Latest consensus state: current_term: 1 leader_uuid: "436a4f1fe35f4a6083332451e97a7626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } } }
I20260806 20:40:46.938692 4856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:46.938376 4855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "436a4f1fe35f4a6083332451e97a7626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } } }
I20260806 20:40:46.939659 4855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:46.945477 4859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:40:46.955804 4859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:40:46.957479 4849 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 20:40:46.993845 4859 catalog_manager.cc:1347] Generated new cluster ID: a1fa4dea5e2d4fffbfd2307cabaaba7f
I20260806 20:40:46.994163 4859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:40:47.053807 4859 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:40:47.057843 4859 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:40:47.075673 4859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Generated new TSK 0
I20260806 20:40:47.077817 4859 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:40:48.438905 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:43657
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:40:48 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786048848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 20:40:48.847450 4894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:48.847937 4894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:48.848145 4894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:48.859946 4894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:48.860159 4894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:48.860404 4894 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:40:48.876194 4894 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.200.20:56417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:48.877920 4894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:48.879665 4894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:48.897614 4905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:48.898218 4906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:48.906765 4894 server_base.cc:1034] running on GCE node
W20260806 20:40:48.907306 4908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:48.908114 4894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:48.909305 4894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:48.910593 4894 hybrid_clock.cc:648] HybridClock initialized: now 1786048848910522 us; error 56 us; skew 500 ppm
I20260806 20:40:48.911032 4894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:48 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:48.915879 4894 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:40:48.929374 4894 webserver.cc:460] Webserver started at http://127.23.200.1:34375/ using document root <none> and password file <none>
I20260806 20:40:48.930464 4894 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:48.930625 4894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:48.931143 4894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:40:48 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:48.951006 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "6ba0ccc8433342f49639c322ba86503b"
format_stamp: "Formatted at 2026-08-06 20:40:48 on dist-test-slave-tm1g"
server_key: "be86836088fc3f15f4a06070fa26582c"
server_key_iv: "49f3db55199769bda5044a8ff7a56cc9"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:48.952214 4894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "6ba0ccc8433342f49639c322ba86503b"
format_stamp: "Formatted at 2026-08-06 20:40:48 on dist-test-slave-tm1g"
server_key: "be86836088fc3f15f4a06070fa26582c"
server_key_iv: "49f3db55199769bda5044a8ff7a56cc9"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:48.959438 4894 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 20:40:48.977922 4916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:48.980067 4894 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:40:48.980391 4894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "6ba0ccc8433342f49639c322ba86503b"
format_stamp: "Formatted at 2026-08-06 20:40:48 on dist-test-slave-tm1g"
server_key: "be86836088fc3f15f4a06070fa26582c"
server_key_iv: "49f3db55199769bda5044a8ff7a56cc9"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:48.980791 4894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:49.007026 4894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:49.011829 4894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:49.012246 4894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:49.013341 4894 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:49.015090 4894 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:49.015251 4894 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 20:40:49.015465 4894 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:49.015600 4894 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:49.104300 4894 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:45195
I20260806 20:40:49.104323 5030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:45195 every 8 connection(s)
I20260806 20:40:49.105893 4894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:40:49.115948 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 4894
I20260806 20:40:49.116176 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:40:49.117287 24352 external_mini_cluster.cc:1428] Setting key 94aca94aa2d6153fde8a4a5ad00c7206
Aug 06 20:40:49 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:49.161041 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:49.113227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:35163 (local address 127.23.200.62:43657)
0806 20:40:49.128763 (+ 15536us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:49.128774 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:49.130031 (+ 1257us) server_negotiation.cc:407] Connection header received
0806 20:40:49.132680 (+ 2649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:49.132687 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:49.132794 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:49.132917 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:49.138106 (+ 5189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:49.139291 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:49.141651 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:49.142272 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:49.142392 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:49.150076 (+ 7684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:49.150084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:49.150089 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:49.150131 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:49.153864 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:49.154928 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:49.154935 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:49.154939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:49.155101 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:49.155905 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:49.155910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:49.155914 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:49.156257 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:49.156377 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:49.160382 (+ 4005us) server_negotiation.cc:299] Negotiation successful
0806 20:40:49.160622 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15344,"thread_start_us":15541,"threads_started":1}
I20260806 20:40:49.163524 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:49.113934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:43657 (local address 127.23.200.1:35163)
0806 20:40:49.129527 (+ 15593us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:49.129583 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:49.132315 (+ 2732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:49.136320 (+ 4005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:49.136336 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:49.137022 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:49.137875 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:49.137914 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:49.139519 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:49.139527 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:49.141480 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:49.141495 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:49.142773 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:49.142783 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:49.143078 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:49.143871 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:49.143925 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:49.149788 (+ 5863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:49.154083 (+ 4295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:49.154093 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:49.154111 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:49.154783 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:49.155256 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:49.155261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:49.155265 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:49.155509 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:49.156485 (+ 976us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:49.156494 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:49.156933 (+ 439us) client_negotiation.cc:769] Sending connection context
0806 20:40:49.161294 (+ 4361us) client_negotiation.cc:240] Negotiation successful
0806 20:40:49.161862 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15222,"thread_start_us":14988,"threads_started":1}
I20260806 20:40:49.165333 5031 heartbeater.cc:347] Connected to a master server at 127.23.200.62:43657
I20260806 20:40:49.165673 5031 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:49.166579 5031 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
Aug 06 20:40:49 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:49.169111 4812 ts_manager.cc:194] Registered new tserver with Master: 6ba0ccc8433342f49639c322ba86503b (127.23.200.1:45195)
I20260806 20:40:49.171274 4812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:35163
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:40:49.219841 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:43657
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:40:49 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048848, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
W20260806 20:40:49.678239 5038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:49.678889 5038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:49.679250 5038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:49.694295 5038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:49.694618 5038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:49.694950 5038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:40:49.708815 5038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.200.20:56417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:49.710222 5038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:49.711573 5038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:49.731626 5045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:49.732977 5046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:49.741070 5048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:49.746706 5038 server_base.cc:1034] running on GCE node
I20260806 20:40:49.747799 5038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:49.754644 5038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:49.758085 5038 hybrid_clock.cc:648] HybridClock initialized: now 1786048849757961 us; error 112 us; skew 500 ppm
I20260806 20:40:49.758543 5038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:49 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048849, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:49.763347 5038 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:40:49.766660 5038 webserver.cc:460] Webserver started at http://127.23.200.2:36301/ using document root <none> and password file <none>
I20260806 20:40:49.768013 5038 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:49.768261 5038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:49.768729 5038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:40:49 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048849, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:49.793232 5038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "65fe97b240f44e288bfa4c436c793a43"
format_stamp: "Formatted at 2026-08-06 20:40:49 on dist-test-slave-tm1g"
server_key: "829007bc73a09c4209254fdcbef7b3ea"
server_key_iv: "894c42c8f9dc818a68db1cea02068c5f"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:49.795217 5038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "65fe97b240f44e288bfa4c436c793a43"
format_stamp: "Formatted at 2026-08-06 20:40:49 on dist-test-slave-tm1g"
server_key: "829007bc73a09c4209254fdcbef7b3ea"
server_key_iv: "894c42c8f9dc818a68db1cea02068c5f"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:49.804486 5038 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.003s sys 0.004s
I20260806 20:40:49.826836 5063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:49.828610 5038 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:40:49.828876 5038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "65fe97b240f44e288bfa4c436c793a43"
format_stamp: "Formatted at 2026-08-06 20:40:49 on dist-test-slave-tm1g"
server_key: "829007bc73a09c4209254fdcbef7b3ea"
server_key_iv: "894c42c8f9dc818a68db1cea02068c5f"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:49.829238 5038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:49.884374 5038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:49.892135 5038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:49.892812 5038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:49.894932 5038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:49.897493 5038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:49.897675 5038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:49.897883 5038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:49.897971 5038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:49.999847 5038 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:32859
I20260806 20:40:49.999864 5180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:32859 every 8 connection(s)
I20260806 20:40:50.001612 5038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:40:50.007984 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5038
I20260806 20:40:50.008199 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:40:50.008872 24352 external_mini_cluster.cc:1428] Setting key a8ba2d96598ab668230f65f694dd99c0
Aug 06 20:40:50 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048849, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:50.044315 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:50.006839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:46375 (local address 127.23.200.62:43657)
0806 20:40:50.020222 (+ 13383us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:50.020228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:50.020244 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:40:50.020295 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:50.020300 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:50.020401 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:50.020514 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:50.023533 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.025787 (+ 2254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.029332 (+ 3545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.030143 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.030239 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:50.035681 (+ 5442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:50.035688 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:50.035693 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:50.035800 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:50.040013 (+ 4213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.040994 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.041000 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.041004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.041167 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.041899 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.041905 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.041909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.042342 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:50.042521 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:50.043534 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0806 20:40:50.043760 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13249,"thread_start_us":13151,"threads_started":1}
I20260806 20:40:50.047041 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:50.006213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:43657 (local address 127.23.200.2:46375)
0806 20:40:50.015273 (+ 9060us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:50.015323 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:50.019279 (+ 3956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:50.020959 (+ 1680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:50.020974 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:50.021740 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:50.023039 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:50.023116 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.026115 (+ 2999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.026122 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:50.029171 (+ 3049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:50.029184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.030645 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.030652 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:50.030927 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:50.031744 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:50.031774 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:50.035431 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:50.040179 (+ 4748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:50.040188 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:50.040206 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:50.040843 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:50.041324 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:50.041329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:50.041333 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:50.041745 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:50.042617 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:50.042625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:50.043057 (+ 432us) client_negotiation.cc:769] Sending connection context
0806 20:40:50.044662 (+ 1605us) client_negotiation.cc:240] Negotiation successful
0806 20:40:50.045275 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8694,"thread_start_us":8338,"threads_started":1}
I20260806 20:40:50.048851 5181 heartbeater.cc:347] Connected to a master server at 127.23.200.62:43657
I20260806 20:40:50.049219 5181 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:50.050329 5181 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
I20260806 20:40:50.052211 4812 ts_manager.cc:194] Registered new tserver with Master: 65fe97b240f44e288bfa4c436c793a43 (127.23.200.2:32859)
I20260806 20:40:50.053377 4812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:46375
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:40:50.098366 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:43657
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:40:50.174991 5031 heartbeater.cc:503] Master 127.23.200.62:43657 was elected leader, sending a full tablet report...
W20260806 20:40:50.583753 5187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:50.584101 5187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:50.584256 5187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:50.594173 5187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:50.594383 5187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:50.594592 5187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:40:50.605566 5187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.200.20:56417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:50.606830 5187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:50.608130 5187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:50.623430 5197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:50.623558 5193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:50.623554 5194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:50.624838 5187 server_base.cc:1034] running on GCE node
I20260806 20:40:50.625453 5187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:50.626430 5187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:50.627667 5187 hybrid_clock.cc:648] HybridClock initialized: now 1786048850627623 us; error 32 us; skew 500 ppm
I20260806 20:40:50.627979 5187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:50 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:50.631209 5187 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:40:50.633062 5187 webserver.cc:460] Webserver started at http://127.23.200.3:38049/ using document root <none> and password file <none>
I20260806 20:40:50.633822 5187 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:50.634047 5187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:50.634558 5187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 20:40:50 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:50.651690 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "3a4d90d22dd14a56be9deccb01b7b36e"
format_stamp: "Formatted at 2026-08-06 20:40:50 on dist-test-slave-tm1g"
server_key: "17847bf9c8aa003f707f2c7aa5e80c91"
server_key_iv: "ca71a963246ba053e9574ed37068d538"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:50.652642 5187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "3a4d90d22dd14a56be9deccb01b7b36e"
format_stamp: "Formatted at 2026-08-06 20:40:50 on dist-test-slave-tm1g"
server_key: "17847bf9c8aa003f707f2c7aa5e80c91"
server_key_iv: "ca71a963246ba053e9574ed37068d538"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:50.658903 5187 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260806 20:40:50.675338 5205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:50.677057 5187 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:40:50.677235 5187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "3a4d90d22dd14a56be9deccb01b7b36e"
format_stamp: "Formatted at 2026-08-06 20:40:50 on dist-test-slave-tm1g"
server_key: "17847bf9c8aa003f707f2c7aa5e80c91"
server_key_iv: "ca71a963246ba053e9574ed37068d538"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:50.677472 5187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:50.702643 5187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:50.707279 5187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:50.707546 5187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:50.708716 5187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:50.710434 5187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:50.710572 5187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:50.710703 5187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:50.710791 5187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:50.761464 5187 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:34775
I20260806 20:40:50.761463 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:34775 every 8 connection(s)
I20260806 20:40:50.762957 5187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:40:50.771201 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5187
I20260806 20:40:50.771389 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:40:50.772147 24352 external_mini_cluster.cc:1428] Setting key 3dae51d3e2802a155a5506508fc226bb
Aug 06 20:40:50 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048850, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:50.786375 5323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:50.765967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:48493 (local address 127.23.200.62:43657)
0806 20:40:50.766688 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:50.766696 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:50.767976 (+ 1280us) server_negotiation.cc:407] Connection header received
0806 20:40:50.769937 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:50.769942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:50.770022 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:50.770154 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:50.772127 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.773157 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.775166 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.775985 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.776075 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:50.780413 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:50.780418 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:50.780421 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:50.780452 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:50.783290 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.784063 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.784068 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.784070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.784187 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.784611 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.784614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.784616 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.784875 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:50.784984 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:50.785781 (+ 797us) server_negotiation.cc:299] Negotiation successful
0806 20:40:50.786015 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":381,"threads_started":1}
I20260806 20:40:50.787701 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:50.766728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:43657 (local address 127.23.200.3:48493)
0806 20:40:50.767811 (+ 1083us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:50.767853 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:50.769695 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:50.770446 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:50.770455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:50.770957 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:50.771929 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:50.771956 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.773302 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.773307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:50.774923 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:50.774958 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.776330 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.776340 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:50.776762 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:50.777609 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:50.777643 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:50.780191 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:50.783441 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:50.783447 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:50.783458 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:50.783953 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:50.784277 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:50.784280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:50.784283 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:50.784508 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:50.784987 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:50.784992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:50.785365 (+ 373us) client_negotiation.cc:769] Sending connection context
0806 20:40:50.785655 (+ 290us) client_negotiation.cc:240] Negotiation successful
0806 20:40:50.786109 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":793,"thread_start_us":555,"threads_started":1}
I20260806 20:40:50.789127 5321 heartbeater.cc:347] Connected to a master server at 127.23.200.62:43657
I20260806 20:40:50.789459 5321 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:50.790199 5321 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
I20260806 20:40:50.791756 4812 ts_manager.cc:194] Registered new tserver with Master: 3a4d90d22dd14a56be9deccb01b7b36e (127.23.200.3:34775)
I20260806 20:40:50.792749 4812 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:48493
I20260806 20:40:50.800379 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:40:50.802058 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 4894
Aug 06 20:40:50 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:40:50.821997 5326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:50.803203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.23.200.1:45195)
0806 20:40:50.804234 (+ 1031us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:50.804241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:50.804265 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:40:50.804322 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:50.804328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:50.804540 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:50.804664 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:50.806290 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.807633 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.809161 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.809817 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.809919 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:50.814861 (+ 4942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:50.814874 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:50.814888 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:50.814923 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:50.818476 (+ 3553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.819499 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.819507 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.819509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.819758 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.820159 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.820164 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.820173 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.820611 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:50.821050 (+ 439us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:50.821316 (+ 266us) server_negotiation.cc:299] Negotiation successful
0806 20:40:50.821630 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":666,"threads_started":1}
I20260806 20:40:50.823339 5026 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:50.906663 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 4894
I20260806 20:40:50.925091 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5038
Aug 06 20:40:50 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:40:50.942530 5328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:50.926004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.23.200.2:32859)
0806 20:40:50.926622 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:50.926629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:50.926652 (+ 23us) server_negotiation.cc:407] Connection header received
0806 20:40:50.926726 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:50.926732 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:50.926940 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:50.927120 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:50.928649 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.929679 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.931309 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:50.932060 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:50.932194 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:50.936908 (+ 4714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:50.936920 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:50.936934 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:50.936967 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:50.940101 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.940696 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.940702 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.940704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.940832 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:50.941156 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:50.941159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:50.941161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:50.941489 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:50.941707 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:50.941960 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 20:40:50.942161 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":318,"threads_started":1}
I20260806 20:40:50.943806 5176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:51.026484 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5038
I20260806 20:40:51.043957 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5187
Aug 06 20:40:51 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:40:51.060400 5330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:51.044798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56424 (local address 127.23.200.3:34775)
0806 20:40:51.045419 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:51.045426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:51.045456 (+ 30us) server_negotiation.cc:407] Connection header received
0806 20:40:51.045523 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:51.045529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:51.045741 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:51.045923 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:51.047448 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:51.048484 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:51.049758 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:51.050475 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:51.050624 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:51.055134 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:51.055144 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:51.055158 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:51.055190 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:51.057981 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:51.058675 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:51.058682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:51.058684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:51.058809 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:51.059142 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:51.059145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:51.059147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:51.059463 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:51.059700 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:51.059928 (+ 228us) server_negotiation.cc:299] Negotiation successful
0806 20:40:51.060116 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":285,"threads_started":1}
I20260806 20:40:51.061615 5316 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:51.144129 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5187
I20260806 20:40:51.161754 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 4779
I20260806 20:40:51.163108 4844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:51.270669 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 4779
I20260806 20:40:51.297982 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:43657
--webserver_interface=127.23.200.62
--webserver_port=46535
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:43657
--ranger_config_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:40:51.619696 5334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:51.619951 5334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:51.620105 5334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:51.627240 5334 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:40:51.627337 5334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:51.627435 5334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:51.627544 5334 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:40:51.627631 5334 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:40:51.637247 5334 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:43657
--ranger_config_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=46535
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:51.638501 5334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:51.639803 5334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:51.651665 5340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:51.651614 5343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:51.651850 5341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:51.652791 5334 server_base.cc:1034] running on GCE node
I20260806 20:40:51.653491 5334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:51.654982 5334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:51.656222 5334 hybrid_clock.cc:648] HybridClock initialized: now 1786048851656155 us; error 52 us; skew 500 ppm
I20260806 20:40:51.656670 5334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:51 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048851, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:51.662562 5334 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:40:51.663921 5334 webserver.cc:460] Webserver started at http://127.23.200.62:46535/ using document root <none> and password file <none>
I20260806 20:40:51.664566 5334 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:51.664713 5334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 20:40:51 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048851, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:51.755219 5334 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:40:51.758244 5351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:51.759632 5334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:40:51.759826 5334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "436a4f1fe35f4a6083332451e97a7626"
format_stamp: "Formatted at 2026-08-06 20:40:46 on dist-test-slave-tm1g"
server_key: "8a9f1fece5fac304c82da85f0147e026"
server_key_iv: "dd2ba6b3c3efac7462213dd5bc167b31"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:51.760797 5334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:51.777943 5334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:51.782882 5334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:51.783219 5334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:51.801606 5334 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:43657
I20260806 20:40:51.801631 5402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:43657 every 8 connection(s)
I20260806 20:40:51.803257 5334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:40:51.809963 5403 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:40:51.813058 5403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Bootstrap starting.
I20260806 20:40:51.813236 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5334
I20260806 20:40:51.814085 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:45195
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:43657
--webserver_port=34375
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:40:51.831746 5403 log.cc:824] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:51.840304 5403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:40:51.840831 5403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Bootstrap complete.
I20260806 20:40:51.850010 5403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:51.850930 5403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436a4f1fe35f4a6083332451e97a7626, State: Initialized, Role: FOLLOWER
I20260806 20:40:51.851532 5403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:51.851781 5403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:40:51.851917 5403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:40:51.852088 5403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:40:51.855856 5403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:51.856328 5403 leader_election.cc:302] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 436a4f1fe35f4a6083332451e97a7626; no voters:
I20260806 20:40:51.856966 5403 leader_election.cc:288] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 20:40:51.857121 5407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:40:51.858289 5407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [term 2 LEADER]: Becoming Leader. State: Replica: 436a4f1fe35f4a6083332451e97a7626, State: Running, Role: LEADER
I20260806 20:40:51.859019 5403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:40:51.859015 5407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } }
I20260806 20:40:51.861728 5409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 436a4f1fe35f4a6083332451e97a7626. Latest consensus state: current_term: 2 leader_uuid: "436a4f1fe35f4a6083332451e97a7626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } } }
I20260806 20:40:51.862172 5409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:51.862808 5408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "436a4f1fe35f4a6083332451e97a7626" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "436a4f1fe35f4a6083332451e97a7626" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 43657 } } }
I20260806 20:40:51.863142 5408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:51.871575 5414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:40:51.877156 5414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:40:51.878569 5414 catalog_manager.cc:1259] Loaded cluster ID: a1fa4dea5e2d4fffbfd2307cabaaba7f
I20260806 20:40:51.878715 5414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:40:51.882656 5414 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:40:51.884097 5414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Loaded TSK: 0
I20260806 20:40:51.885624 5414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:40:51.891368 5403 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 20:40:52.265251 5405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:52.265851 5405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:52.266077 5405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:52.277612 5405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:52.277801 5405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:52.278035 5405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:40:52.293169 5405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:45195
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=34375
--tserver_master_addrs=127.23.200.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:52.294952 5405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:52.296298 5405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:52.334853 5442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:52.337136 5443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:52.337666 5405 server_base.cc:1034] running on GCE node
W20260806 20:40:52.338442 5445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:52.339201 5405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:52.341295 5405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:52.356976 5405 hybrid_clock.cc:648] HybridClock initialized: now 1786048852356884 us; error 90 us; skew 500 ppm
I20260806 20:40:52.357528 5405 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:52 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048852, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:52.361604 5405 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:40:52.374266 5405 webserver.cc:460] Webserver started at http://127.23.200.1:34375/ using document root <none> and password file <none>
I20260806 20:40:52.375375 5405 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:52.375634 5405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 20:40:52 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048852, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:52.412540 5405 fs_manager.cc:709] Time spent opening directory manager: real 0.013s user 0.004s sys 0.000s
I20260806 20:40:52.427007 5453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:52.429628 5405 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.012s sys 0.000s
I20260806 20:40:52.429894 5405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "6ba0ccc8433342f49639c322ba86503b"
format_stamp: "Formatted at 2026-08-06 20:40:48 on dist-test-slave-tm1g"
server_key: "be86836088fc3f15f4a06070fa26582c"
server_key_iv: "49f3db55199769bda5044a8ff7a56cc9"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:52.431457 5405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:52.478850 5405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:52.484537 5405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:52.484928 5405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:52.486330 5405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:52.488094 5405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:52.488205 5405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:52.488314 5405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:52.488385 5405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:52.539252 5405 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:45195
I20260806 20:40:52.539304 5568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:45195 every 8 connection(s)
I20260806 20:40:52.540913 5405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:40:52.541630 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5405
I20260806 20:40:52.542496 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:32859
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:43657
--webserver_port=36301
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:40:52 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048852, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:52.597591 5572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:52.544453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:43657 (local address 127.23.200.1:57947)
0806 20:40:52.569234 (+ 24781us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:52.569295 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:52.572353 (+ 3058us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:52.575418 (+ 3065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:52.575432 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:52.576159 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:52.577069 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:52.577100 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:52.579694 (+ 2594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:52.579700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:52.581844 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:52.581860 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:52.583318 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:52.583327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:52.583618 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:52.585613 (+ 1995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:52.585646 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:52.588337 (+ 2691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:52.592774 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:52.592781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:52.592795 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:52.593351 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:52.593899 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:52.593904 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:52.593907 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:52.594147 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:52.595199 (+ 1052us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:52.595206 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:52.595582 (+ 376us) client_negotiation.cc:769] Sending connection context
0806 20:40:52.595879 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 20:40:52.596383 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24403,"thread_start_us":29772,"threads_started":1}
I20260806 20:40:52.597828 5573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:52.545547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:57947 (local address 127.23.200.62:43657)
0806 20:40:52.568336 (+ 22789us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:52.568362 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:52.572982 (+ 4620us) server_negotiation.cc:407] Connection header received
0806 20:40:52.573495 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:52.573520 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:52.574071 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:52.574599 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:52.577342 (+ 2743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:52.579474 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:52.582006 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:52.583049 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:52.583220 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:52.588632 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:52.588642 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:52.588662 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:52.588710 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:52.592602 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:52.593534 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:52.593542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:52.593549 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:52.593756 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:52.594283 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:52.594288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:52.594291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:52.594932 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:52.595113 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:52.596178 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0806 20:40:52.596591 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22439,"thread_start_us":22743,"threads_started":1}
I20260806 20:40:52.601806 5570 heartbeater.cc:347] Connected to a master server at 127.23.200.62:43657
I20260806 20:40:52.602106 5570 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:52.602957 5570 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
W20260806 20:40:52.606233 5570 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:43657 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 20:40:52.606503 5570 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:52.606760 5570 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
I20260806 20:40:52.608108 5570 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:52.608449 5570 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
W20260806 20:40:52.609556 5570 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 20:40:52.959110 5574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:52.959405 5574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:52.959628 5574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:52.967190 5574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:52.967286 5574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:52.967499 5574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:40:52.980970 5574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:32859
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=36301
--tserver_master_addrs=127.23.200.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:52.982781 5574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:52.984419 5574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:53.001206 5590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:53.001246 5589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:53.004029 5593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:53.004601 5574 server_base.cc:1034] running on GCE node
I20260806 20:40:53.005261 5574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:53.006424 5574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:53.007645 5574 hybrid_clock.cc:648] HybridClock initialized: now 1786048853007588 us; error 55 us; skew 500 ppm
I20260806 20:40:53.008013 5574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:53.011253 5574 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:40:53.013104 5574 webserver.cc:460] Webserver started at http://127.23.200.2:36301/ using document root <none> and password file <none>
I20260806 20:40:53.013821 5574 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:53.014000 5574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:53.040242 5574 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 20:40:53.053440 5601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:53.055366 5574 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.009s sys 0.000s
I20260806 20:40:53.055614 5574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "65fe97b240f44e288bfa4c436c793a43"
format_stamp: "Formatted at 2026-08-06 20:40:49 on dist-test-slave-tm1g"
server_key: "829007bc73a09c4209254fdcbef7b3ea"
server_key_iv: "894c42c8f9dc818a68db1cea02068c5f"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:53.057156 5574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:53.096417 5574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:53.102046 5574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:53.102423 5574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:53.103543 5574 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:53.105116 5574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:53.105238 5574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:53.105406 5574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:53.105496 5574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:53.163408 5574 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:32859
I20260806 20:40:53.163432 5717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:32859 every 8 connection(s)
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:53.165719 5574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:40:53.170032 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5574
I20260806 20:40:53.173041 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.200.20:56417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:34775
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:43657
--webserver_port=38049
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:45383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:53.221062 5720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:53.168570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:56583 (local address 127.23.200.62:43657)
0806 20:40:53.198333 (+ 29763us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:53.198342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:53.198371 (+ 29us) server_negotiation.cc:407] Connection header received
0806 20:40:53.198453 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:53.198461 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:53.198581 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:53.198719 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:53.201240 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:53.202523 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:53.205209 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:53.206016 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:53.206150 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:53.212549 (+ 6399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:53.212558 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:53.212566 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:53.212615 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:53.216880 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:53.218051 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:53.218059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:53.218062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:53.218230 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:53.218872 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:53.218877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:53.218880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:53.219240 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:53.219380 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:53.220454 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0806 20:40:53.220693 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29567,"thread_start_us":29452,"threads_started":1}
I20260806 20:40:53.222851 5722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:53.169278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:43657 (local address 127.23.200.2:56583)
0806 20:40:53.194090 (+ 24812us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:53.194158 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:53.197475 (+ 3317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:53.199951 (+ 2476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:53.199960 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:53.200435 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:53.201034 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:53.201052 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:53.202741 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:53.202749 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:53.205011 (+ 2262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:53.205029 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:53.206477 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:53.206484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:53.206810 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:53.207666 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:53.207700 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:53.212282 (+ 4582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:53.217123 (+ 4841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:53.217135 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:53.217158 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:53.217886 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:53.218417 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:53.218425 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:53.218430 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:53.218735 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:53.219419 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:53.219430 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:53.219959 (+ 529us) client_negotiation.cc:769] Sending connection context
0806 20:40:53.220335 (+ 376us) client_negotiation.cc:240] Negotiation successful
0806 20:40:53.221018 (+ 683us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24368,"thread_start_us":24801,"threads_started":1}
I20260806 20:40:53.224503 5718 heartbeater.cc:347] Connected to a master server at 127.23.200.62:43657
I20260806 20:40:53.224835 5718 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:53.225759 5718 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
I20260806 20:40:53.227842 5368 ts_manager.cc:194] Registered new tserver with Master: 65fe97b240f44e288bfa4c436c793a43 (127.23.200.2:32859)
I20260806 20:40:53.229979 5368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:56583
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:53.609714 5570 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:53.610208 5570 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
I20260806 20:40:53.612015 5368 ts_manager.cc:194] Registered new tserver with Master: 6ba0ccc8433342f49639c322ba86503b (127.23.200.1:45195)
I20260806 20:40:53.613581 5368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:57947
W20260806 20:40:53.720336 5723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:53.720829 5723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:53.721084 5723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:53.735826 5723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:53.736018 5723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:53.736241 5723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:40:53.762153 5723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:45383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.200.20:56417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:34775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=38049
--tserver_master_addrs=127.23.200.62:43657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:53.764194 5723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:53.765997 5723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:53.788336 5731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:53.789769 5730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:53.794688 5733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:53.796633 5723 server_base.cc:1034] running on GCE node
I20260806 20:40:53.797601 5723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:53.805364 5723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:53.807255 5723 hybrid_clock.cc:648] HybridClock initialized: now 1786048853807048 us; error 135 us; skew 500 ppm
I20260806 20:40:53.808027 5723 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:53.813944 5723 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:40:53.817374 5723 webserver.cc:460] Webserver started at http://127.23.200.3:38049/ using document root <none> and password file <none>
I20260806 20:40:53.818722 5723 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:53.818966 5723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 20:40:53 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:53.848784 5723 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 20:40:53.853789 5742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:53.855599 5723 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 20:40:53.855906 5723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "3a4d90d22dd14a56be9deccb01b7b36e"
format_stamp: "Formatted at 2026-08-06 20:40:50 on dist-test-slave-tm1g"
server_key: "17847bf9c8aa003f707f2c7aa5e80c91"
server_key_iv: "ca71a963246ba053e9574ed37068d538"
server_key_version: "kuduclusterkey@0"
I20260806 20:40:53.857944 5723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:53.890461 5723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:53.897346 5723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:53.897948 5723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:53.899869 5723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:40:53.903363 5723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:40:53.903589 5723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:53.903752 5723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:40:53.903946 5723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 20:40:54 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for HTTP/127.23.200.20@KRBTEST.COM
I20260806 20:40:54.153563 5723 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:34775
I20260806 20:40:54.155143 5856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:34775 every 8 connection(s)
I20260806 20:40:54.155771 5723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:40:54.163722 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5723
Aug 06 20:40:54 dist-test-slave-tm1g krb5kdc[4323](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048853, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:54.209477 5860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.164971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43636 (local address 127.23.200.62:43657)
0806 20:40:54.167348 (+ 2377us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.167358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.180483 (+ 13125us) server_negotiation.cc:407] Connection header received
0806 20:40:54.180742 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.180748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.180891 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.181040 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:54.189622 (+ 8582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.190831 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.194518 (+ 3687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.195349 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.195599 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.199227 (+ 3628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:54.199234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:54.199239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:54.199295 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:54.203940 (+ 4645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.204939 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.204946 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.204952 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.205122 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.205784 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.205791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.205794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.206166 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:54.206467 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.208632 (+ 2165us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.208938 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":2210,"thread_start_us":2064,"threads_started":1}
I20260806 20:40:54.216958 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.167254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:38699 (local address 127.23.200.62:43657)
0806 20:40:54.171296 (+ 4042us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.171303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.182076 (+ 10773us) server_negotiation.cc:407] Connection header received
0806 20:40:54.186418 (+ 4342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.186424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.186527 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.186647 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:54.195294 (+ 8647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.196693 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.200456 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.201665 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.201767 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.206690 (+ 4923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:54.206695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:54.206698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:54.206727 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:54.210473 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.213688 (+ 3215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.213708 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.213711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.213863 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.214585 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.214591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.214595 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.214944 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:54.215089 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.216124 (+ 1035us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.216364 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3891,"thread_start_us":4775,"threads_started":1}
I20260806 20:40:54.219769 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.166983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:43657 (local address 127.23.200.3:38699)
0806 20:40:54.181862 (+ 14879us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.181943 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.185900 (+ 3957us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.190213 (+ 4313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.190229 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.190952 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:54.192019 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.192051 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.196882 (+ 4831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.196888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.200251 (+ 3363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.200273 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.202326 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.202338 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.202761 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.203672 (+ 911us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:54.203733 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:54.206419 (+ 2686us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:54.211515 (+ 5096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:54.211526 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:54.211547 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:54.213543 (+ 1996us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:54.214044 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:54.214049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:54.214052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:54.214436 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:54.215187 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:54.215195 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:54.215669 (+ 474us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.217534 (+ 1865us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.218180 (+ 646us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14395,"thread_start_us":15289,"threads_started":1}
I20260806 20:40:54.221810 5857 heartbeater.cc:347] Connected to a master server at 127.23.200.62:43657
I20260806 20:40:54.222138 5857 heartbeater.cc:464] Registering TS with master...
I20260806 20:40:54.223145 5857 heartbeater.cc:511] Master 127.23.200.62:43657 requested a full tablet report, sending...
I20260806 20:40:54.224992 5368 ts_manager.cc:194] Registered new tserver with Master: 3a4d90d22dd14a56be9deccb01b7b36e (127.23.200.3:34775)
I20260806 20:40:54.226397 5368 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:38699
I20260806 20:40:54.230254 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:40:54.234642 5718 heartbeater.cc:503] Master 127.23.200.62:43657 was elected leader, sending a full tablet report...
I20260806 20:40:54.260744 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.236018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.23.200.62:43657)
0806 20:40:54.236211 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.236218 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.240426 (+ 4208us) server_negotiation.cc:407] Connection header received
0806 20:40:54.240750 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.240757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.240859 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.240934 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:54.243318 (+ 2384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.244604 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.246859 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.247548 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.247664 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.251530 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:54.251538 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:54.251543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:54.251591 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:54.256463 (+ 4872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.257449 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.257456 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.257460 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.257634 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.258223 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.258236 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.258239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.259024 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:54.259325 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.260096 (+ 771us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.260351 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 20:40:54.267951 5368 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:40:54.281505 5368 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:40:54.341303 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.318918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.23.200.2:32859)
0806 20:40:54.319789 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.319798 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.330666 (+ 10868us) server_negotiation.cc:407] Connection header received
0806 20:40:54.330996 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.331004 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.331216 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.331383 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.332793 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.334686 (+ 1893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.338014 (+ 3328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.339609 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.339705 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.339984 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.340775 (+ 791us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.340948 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":1058,"threads_started":1}
I20260806 20:40:54.341626 5884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.318901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:34775 (local address 127.0.0.1:56432)
0806 20:40:54.320639 (+ 1738us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.320691 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.320918 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.330629 (+ 9711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.330636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.330663 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.331075 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.331089 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.334833 (+ 3744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.334838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.338624 (+ 3786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.338638 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.340568 (+ 1930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.340574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.340766 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.340803 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.340860 (+ 57us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.341051 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1423,"mutex_wait_us":25,"thread_start_us":11324,"threads_started":1}
I20260806 20:40:54.342190 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.320103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56432 (local address 127.23.200.3:34775)
0806 20:40:54.327169 (+ 7066us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.327182 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.327227 (+ 45us) server_negotiation.cc:407] Connection header received
0806 20:40:54.329406 (+ 2179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.329419 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.329701 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.330852 (+ 1151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.331255 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.334633 (+ 3378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.338834 (+ 4201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.340319 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.341265 (+ 946us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.341616 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.341681 (+ 65us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.341830 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6836,"thread_start_us":6606,"threads_started":1}
I20260806 20:40:54.342473 5882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.319783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32859 (local address 127.0.0.1:37362)
0806 20:40:54.330562 (+ 10779us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.330582 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.330769 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.331703 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.331710 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.331736 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.332542 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.332557 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.334844 (+ 2287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.334848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.337832 (+ 2984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.337847 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.340219 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.340226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.340453 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.340546 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.341832 (+ 1286us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.341971 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10649,"mutex_wait_us":61,"thread_start_us":416,"threads_started":1}
I20260806 20:40:54.344260 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.320821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.23.200.1:45195)
0806 20:40:54.328402 (+ 7581us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.328412 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.329114 (+ 702us) server_negotiation.cc:407] Connection header received
0806 20:40:54.329355 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.329364 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.329621 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.329874 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.331557 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.333552 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.338108 (+ 4556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.339967 (+ 1859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.340133 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.340512 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.342226 (+ 1714us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.342466 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7407,"thread_start_us":7236,"threads_started":1}
I20260806 20:40:54.344954 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.319622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33510)
0806 20:40:54.328960 (+ 9338us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.328985 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.329147 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.330188 (+ 1041us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.330196 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.330227 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.331280 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.331290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.333724 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.333729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.337788 (+ 4059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.337814 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.340714 (+ 2900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.340721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.340907 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.340939 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.341021 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.341147 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9183,"mutex_wait_us":3,"thread_start_us":8993,"threads_started":1}
I20260806 20:40:54.347074 5653 tablet_service.cc:1467] Processing CreateTablet for tablet bcf5fdf0c61c4a60981a3975e6edb0b7 (DEFAULT_TABLE table=test-table [id=976dbe63b7fa414888d6aaffa65278ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:40:54.347087 5792 tablet_service.cc:1467] Processing CreateTablet for tablet bcf5fdf0c61c4a60981a3975e6edb0b7 (DEFAULT_TABLE table=test-table [id=976dbe63b7fa414888d6aaffa65278ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:40:54.348841 5792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf5fdf0c61c4a60981a3975e6edb0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:54.349287 5653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf5fdf0c61c4a60981a3975e6edb0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:54.349289 5503 tablet_service.cc:1467] Processing CreateTablet for tablet bcf5fdf0c61c4a60981a3975e6edb0b7 (DEFAULT_TABLE table=test-table [id=976dbe63b7fa414888d6aaffa65278ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:40:54.351089 5503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcf5fdf0c61c4a60981a3975e6edb0b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:54.382261 5888 tablet_bootstrap.cc:492] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Bootstrap starting.
I20260806 20:40:54.388980 5889 tablet_bootstrap.cc:492] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Bootstrap starting.
I20260806 20:40:54.389767 5890 tablet_bootstrap.cc:492] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Bootstrap starting.
I20260806 20:40:54.392608 5890 tablet_bootstrap.cc:654] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:54.392800 5888 tablet_bootstrap.cc:654] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:54.394194 5889 tablet_bootstrap.cc:654] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:54.394204 5890 log.cc:824] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:54.394204 5888 log.cc:824] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:54.395671 5889 log.cc:824] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:54.403918 5890 tablet_bootstrap.cc:492] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: No bootstrap required, opened a new log
I20260806 20:40:54.403929 5888 tablet_bootstrap.cc:492] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: No bootstrap required, opened a new log
I20260806 20:40:54.404215 5890 ts_tablet_manager.cc:1397] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.004s
I20260806 20:40:54.404215 5888 ts_tablet_manager.cc:1397] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.005s
I20260806 20:40:54.411962 5889 tablet_bootstrap.cc:492] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: No bootstrap required, opened a new log
I20260806 20:40:54.412352 5889 ts_tablet_manager.cc:1397] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Time spent bootstrapping tablet: real 0.024s user 0.006s sys 0.015s
I20260806 20:40:54.417685 5888 raft_consensus.cc:348] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.417896 5890 raft_consensus.cc:348] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.418355 5888 raft_consensus.cc:374] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:54.418545 5890 raft_consensus.cc:374] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:54.418612 5888 raft_consensus.cc:729] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a4d90d22dd14a56be9deccb01b7b36e, State: Initialized, Role: FOLLOWER
I20260806 20:40:54.418745 5890 raft_consensus.cc:729] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65fe97b240f44e288bfa4c436c793a43, State: Initialized, Role: FOLLOWER
I20260806 20:40:54.419543 5888 consensus_queue.cc:260] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.419572 5890 consensus_queue.cc:260] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.426791 5889 raft_consensus.cc:348] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.427309 5889 raft_consensus.cc:374] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:54.427459 5889 raft_consensus.cc:729] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ba0ccc8433342f49639c322ba86503b, State: Initialized, Role: FOLLOWER
I20260806 20:40:54.428390 5890 ts_tablet_manager.cc:1428] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Time spent starting tablet: real 0.024s user 0.015s sys 0.007s
I20260806 20:40:54.428385 5889 consensus_queue.cc:260] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.430388 5857 heartbeater.cc:503] Master 127.23.200.62:43657 was elected leader, sending a full tablet report...
I20260806 20:40:54.431355 5570 heartbeater.cc:503] Master 127.23.200.62:43657 was elected leader, sending a full tablet report...
I20260806 20:40:54.432046 5889 ts_tablet_manager.cc:1428] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Time spent starting tablet: real 0.020s user 0.018s sys 0.000s
I20260806 20:40:54.430410 5888 ts_tablet_manager.cc:1428] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Time spent starting tablet: real 0.026s user 0.014s sys 0.011s
W20260806 20:40:54.445420 5858 tablet.cc:2369] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:40:54.452078 5719 tablet.cc:2369] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:40:54.564985 5571 tablet.cc:2369] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:40:54.696856 5895 raft_consensus.cc:482] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:40:54.697142 5895 raft_consensus.cc:504] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.698915 5895 leader_election.cc:288] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a4d90d22dd14a56be9deccb01b7b36e (127.23.200.3:34775), 6ba0ccc8433342f49639c322ba86503b (127.23.200.1:45195)
I20260806 20:40:54.702252 5894 raft_consensus.cc:482] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:40:54.702754 5894 raft_consensus.cc:504] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.710480 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.699453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:50491 (local address 127.23.200.1:45195)
0806 20:40:54.699727 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.699736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.700671 (+ 935us) server_negotiation.cc:407] Connection header received
0806 20:40:54.700785 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.700790 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.700864 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.700965 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.701736 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.703265 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.707429 (+ 4164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.709142 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.709270 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.709351 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.710034 (+ 683us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.710172 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260806 20:40:54.710649 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.699454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:43605 (local address 127.23.200.3:34775)
0806 20:40:54.699681 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.699686 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.700552 (+ 866us) server_negotiation.cc:407] Connection header received
0806 20:40:54.700779 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.700785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.700864 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.700965 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.702647 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.704193 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.707898 (+ 3705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.709454 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.709582 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.709646 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.710253 (+ 607us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.710407 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:40:54.711320 5897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.699262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:34775 (local address 127.23.200.2:43605)
0806 20:40:54.700428 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.700448 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.700620 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.701759 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.701765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.701798 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.702443 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.702459 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.704391 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.704403 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.707663 (+ 3260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.707686 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.709883 (+ 2197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.709889 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.710054 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.710083 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.710796 (+ 713us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.710935 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"thread_start_us":506,"threads_started":1}
I20260806 20:40:54.712054 5898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.699364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.23.200.2:50491)
0806 20:40:54.700570 (+ 1206us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.700592 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.700697 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.701003 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.701011 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.701048 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.701552 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.701569 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.703453 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.703458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.707258 (+ 3800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.707271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.709629 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.709636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.709818 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.709853 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.711366 (+ 1513us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.711503 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"mutex_wait_us":3,"thread_start_us":709,"threads_started":1}
I20260806 20:40:54.712797 5812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf5fdf0c61c4a60981a3975e6edb0b7" candidate_uuid: "65fe97b240f44e288bfa4c436c793a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" is_pre_election: true
I20260806 20:40:54.713052 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf5fdf0c61c4a60981a3975e6edb0b7" candidate_uuid: "65fe97b240f44e288bfa4c436c793a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba0ccc8433342f49639c322ba86503b" is_pre_election: true
I20260806 20:40:54.713402 5812 raft_consensus.cc:2454] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fe97b240f44e288bfa4c436c793a43 in term 0.
I20260806 20:40:54.713703 5523 raft_consensus.cc:2454] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65fe97b240f44e288bfa4c436c793a43 in term 0.
I20260806 20:40:54.714258 5602 leader_election.cc:302] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a4d90d22dd14a56be9deccb01b7b36e, 65fe97b240f44e288bfa4c436c793a43; no voters:
I20260806 20:40:54.714781 5895 raft_consensus.cc:2781] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:40:54.714964 5895 raft_consensus.cc:482] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:40:54.715094 5895 raft_consensus.cc:3037] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:54.717681 5895 raft_consensus.cc:504] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.718022 5894 leader_election.cc:288] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 65fe97b240f44e288bfa4c436c793a43 (127.23.200.2:32859), 6ba0ccc8433342f49639c322ba86503b (127.23.200.1:45195)
I20260806 20:40:54.718839 5812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf5fdf0c61c4a60981a3975e6edb0b7" candidate_uuid: "65fe97b240f44e288bfa4c436c793a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a4d90d22dd14a56be9deccb01b7b36e"
I20260806 20:40:54.718995 5895 leader_election.cc:288] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [CANDIDATE]: Term 1 election: Requested vote from peers 3a4d90d22dd14a56be9deccb01b7b36e (127.23.200.3:34775), 6ba0ccc8433342f49639c322ba86503b (127.23.200.1:45195)
I20260806 20:40:54.719064 5812 raft_consensus.cc:3037] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:54.719276 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf5fdf0c61c4a60981a3975e6edb0b7" candidate_uuid: "65fe97b240f44e288bfa4c436c793a43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba0ccc8433342f49639c322ba86503b"
I20260806 20:40:54.719561 5523 raft_consensus.cc:3037] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:54.720911 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.705141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32859 (local address 127.23.200.3:45707)
0806 20:40:54.705547 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.705564 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.705665 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.706902 (+ 1237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.706906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.706930 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.707258 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.707271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.709102 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.709108 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.711302 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.711316 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.719687 (+ 8371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.719693 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.719967 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.720010 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.720187 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.720370 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287}
I20260806 20:40:54.721504 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.706390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:45707 (local address 127.23.200.2:32859)
0806 20:40:54.706637 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.706645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.706665 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:40:54.706719 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.706723 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.706787 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.706882 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.707421 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.708883 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.711829 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.719456 (+ 7627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.719679 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.719750 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.721197 (+ 1447us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.721290 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260806 20:40:54.722285 5673 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf5fdf0c61c4a60981a3975e6edb0b7" candidate_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "65fe97b240f44e288bfa4c436c793a43" is_pre_election: true
I20260806 20:40:54.722774 5812 raft_consensus.cc:2454] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fe97b240f44e288bfa4c436c793a43 in term 1.
I20260806 20:40:54.722903 5673 raft_consensus.cc:2379] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a4d90d22dd14a56be9deccb01b7b36e in current term 1: Already voted for candidate 65fe97b240f44e288bfa4c436c793a43 in this term.
I20260806 20:40:54.723242 5602 leader_election.cc:302] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a4d90d22dd14a56be9deccb01b7b36e, 65fe97b240f44e288bfa4c436c793a43; no voters:
I20260806 20:40:54.723542 5895 raft_consensus.cc:2781] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:40:54.724376 5895 raft_consensus.cc:686] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 1 LEADER]: Becoming Leader. State: Replica: 65fe97b240f44e288bfa4c436c793a43, State: Running, Role: LEADER
I20260806 20:40:54.724848 5895 consensus_queue.cc:237] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } }
I20260806 20:40:54.725679 5523 raft_consensus.cc:2454] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65fe97b240f44e288bfa4c436c793a43 in term 1.
I20260806 20:40:54.729943 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.706073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:46135 (local address 127.23.200.1:45195)
0806 20:40:54.710694 (+ 4621us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.710699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.710715 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:40:54.710784 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.710789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.710852 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.710936 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:40:54.721023 (+ 10087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.725511 (+ 4488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.728036 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.728882 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.729073 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.729140 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.729431 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.729560 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4530,"thread_start_us":12767,"threads_started":1}
I20260806 20:40:54.729984 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.705431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.23.200.3:46135)
0806 20:40:54.706426 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:54.706444 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:54.706584 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:54.719831 (+ 13247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:54.719836 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:54.719864 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:40:54.720749 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.720763 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.725664 (+ 4901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.725669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.727874 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:54.727887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.729088 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.729094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:54.729276 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.729298 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:40:54.729392 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 20:40:54.729528 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":887,"thread_start_us":997,"threads_started":1}
I20260806 20:40:54.730459 5368 catalog_manager.cc:5495] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65fe97b240f44e288bfa4c436c793a43 (127.23.200.2). New cstate: current_term: 1 leader_uuid: "65fe97b240f44e288bfa4c436c793a43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "65fe97b240f44e288bfa4c436c793a43" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32859 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 } health_report { overall_health: UNKNOWN } } }
I20260806 20:40:54.731101 5523 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcf5fdf0c61c4a60981a3975e6edb0b7" candidate_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ba0ccc8433342f49639c322ba86503b" is_pre_election: true
I20260806 20:40:54.731451 5523 raft_consensus.cc:2379] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3a4d90d22dd14a56be9deccb01b7b36e in current term 1: Already voted for candidate 65fe97b240f44e288bfa4c436c793a43 in this term.
I20260806 20:40:54.732192 5747 leader_election.cc:302] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3a4d90d22dd14a56be9deccb01b7b36e; no voters: 65fe97b240f44e288bfa4c436c793a43, 6ba0ccc8433342f49639c322ba86503b
I20260806 20:40:54.732575 5894 raft_consensus.cc:2726] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:40:54.791828 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.785194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37374 (local address 127.23.200.2:32859)
0806 20:40:54.785454 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.785461 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.785663 (+ 202us) server_negotiation.cc:407] Connection header received
0806 20:40:54.785922 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.785927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.786021 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.786180 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:40:54.786813 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.787594 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.789134 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.789571 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.789717 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.790271 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:40:54.791087 (+ 816us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:40:54.791242 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.791399 (+ 157us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.791490 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 20:40:54.800906 5523 raft_consensus.cc:1261] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 1 FOLLOWER]: Refusing update from remote peer 65fe97b240f44e288bfa4c436c793a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:40:54.800964 5812 raft_consensus.cc:1261] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 1 FOLLOWER]: Refusing update from remote peer 65fe97b240f44e288bfa4c436c793a43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:40:54.802059 5901 consensus_queue.cc:1035] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ba0ccc8433342f49639c322ba86503b" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:40:54.802824 5895 consensus_queue.cc:1035] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a4d90d22dd14a56be9deccb01b7b36e" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 34775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:40:54.819157 5905 mvcc.cc:205] Tried to move back new op lower bound from 7315656109255000064 to 7315656108957388800. Current Snapshot: MvccSnapshot[applied={T|T < 7315656109255000064}]
I20260806 20:40:54.824761 5906 mvcc.cc:205] Tried to move back new op lower bound from 7315656109255000064 to 7315656108957388800. Current Snapshot: MvccSnapshot[applied={T|T < 7315656109255000064}]
I20260806 20:40:54.835753 5907 mvcc.cc:205] Tried to move back new op lower bound from 7315656109255000064 to 7315656108957388800. Current Snapshot: MvccSnapshot[applied={T|T < 7315656109255000064}]
I20260806 20:40:54.855751 5366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:40:54.856084 5366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:40:54.860189 5366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 436a4f1fe35f4a6083332451e97a7626: Sending DeleteTablet for 3 replicas of tablet bcf5fdf0c61c4a60981a3975e6edb0b7
I20260806 20:40:54.861275 5792 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf5fdf0c61c4a60981a3975e6edb0b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:40:54 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56432
I20260806 20:40:54.861310 5653 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf5fdf0c61c4a60981a3975e6edb0b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:40:54 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:37362
I20260806 20:40:54.861660 5503 tablet_service.cc:1514] Processing DeleteTablet for tablet bcf5fdf0c61c4a60981a3975e6edb0b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:40:54 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:33510
I20260806 20:40:54.864135 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5405
I20260806 20:40:54.875782 5917 tablet_replica.cc:330] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: stopping tablet replica
I20260806 20:40:54.876503 5917 raft_consensus.cc:2229] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:40:54.878989 5916 tablet_replica.cc:330] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: stopping tablet replica
I20260806 20:40:54.879456 5917 raft_consensus.cc:2258] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:40:54.880404 5918 tablet_replica.cc:330] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: stopping tablet replica
I20260806 20:40:54.880844 5918 raft_consensus.cc:2229] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:40:54.881009 5916 raft_consensus.cc:2229] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:40:54.881656 5916 raft_consensus.cc:2258] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:40:54.881667 5918 raft_consensus.cc:2258] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:40:54.882431 5917 ts_tablet_manager.cc:1905] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:40:54.883985 5918 ts_tablet_manager.cc:1905] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:40:54.884519 5916 ts_tablet_manager.cc:1905] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:40:54.893162 5916 ts_tablet_manager.cc:1918] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:40:54.893410 5916 log.cc:1196] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/bcf5fdf0c61c4a60981a3975e6edb0b7
I20260806 20:40:54.894008 5916 ts_tablet_manager.cc:1939] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 3a4d90d22dd14a56be9deccb01b7b36e: Deleting consensus metadata
I20260806 20:40:54.896025 5352 catalog_manager.cc:4841] TS 3a4d90d22dd14a56be9deccb01b7b36e (127.23.200.3:34775): tablet bcf5fdf0c61c4a60981a3975e6edb0b7 (table test-table [id=976dbe63b7fa414888d6aaffa65278ce]) successfully deleted
I20260806 20:40:54.896665 5917 ts_tablet_manager.cc:1918] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:40:54.897290 5917 log.cc:1196] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/bcf5fdf0c61c4a60981a3975e6edb0b7
I20260806 20:40:54.899513 5887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:54.865508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33526 (local address 127.23.200.1:45195)
0806 20:40:54.865738 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:54.865748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:54.874044 (+ 8296us) server_negotiation.cc:407] Connection header received
0806 20:40:54.874251 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:54.874259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:54.874407 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:54.874524 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:54.876743 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.878354 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.884736 (+ 6382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:54.885446 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:54.885583 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:54.889717 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:54.889730 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:54.889748 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:54.889814 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:54.895607 (+ 5793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.896477 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.896486 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.896489 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.896713 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:54.897122 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:54.897128 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:54.897132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:54.897959 (+ 827us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:54.898324 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:54.898646 (+ 322us) server_negotiation.cc:299] Negotiation successful
0806 20:40:54.898984 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 20:40:54.899603 5917 ts_tablet_manager.cc:1939] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 65fe97b240f44e288bfa4c436c793a43: Deleting consensus metadata
I20260806 20:40:54.899657 5918 ts_tablet_manager.cc:1918] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:40:54.900117 5918 log.cc:1196] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/bcf5fdf0c61c4a60981a3975e6edb0b7
I20260806 20:40:54.900799 5564 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:54.901532 5353 catalog_manager.cc:4841] TS 65fe97b240f44e288bfa4c436c793a43 (127.23.200.2:32859): tablet bcf5fdf0c61c4a60981a3975e6edb0b7 (table test-table [id=976dbe63b7fa414888d6aaffa65278ce]) successfully deleted
I20260806 20:40:54.992758 5918 ts_tablet_manager.cc:1939] T bcf5fdf0c61c4a60981a3975e6edb0b7 P 6ba0ccc8433342f49639c322ba86503b: Deleting consensus metadata
I20260806 20:40:54.993016 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5405
W20260806 20:40:55.012431 5355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:45195 (error 108)
I20260806 20:40:55.012600 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5574
I20260806 20:40:55.015134 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.014213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33532)
0806 20:40:55.014754 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.014910 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":263,"threads_started":1}
W20260806 20:40:55.015456 5355 catalog_manager.cc:4584] TS 6ba0ccc8433342f49639c322ba86503b (127.23.200.1:45195): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bcf5fdf0c61c4a60981a3975e6edb0b7: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
I20260806 20:40:55.027667 5881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.013417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.23.200.2:32859)
0806 20:40:55.013632 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:55.013638 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:55.013728 (+ 90us) server_negotiation.cc:407] Connection header received
0806 20:40:55.013909 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:55.013916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:55.014005 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:55.014117 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:55.015836 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:55.016600 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:55.017942 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:55.018613 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:55.018769 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:55.021342 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:55.021364 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:55.021379 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:55.021409 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:55.024472 (+ 3063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:55.025212 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:55.025221 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:55.025224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:55.025377 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:55.025765 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:55.025771 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:55.025774 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:55.026200 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:55.026557 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:55.026949 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 20:40:55.027179 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:40:55.028393 5713 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:55.069149 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.068716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33534)
0806 20:40:55.068915 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.069005 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 20:40:55.070235 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.069880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33544)
0806 20:40:55.070047 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.070113 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 20:40:55.130623 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.130147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33552)
0806 20:40:55.130390 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.130479 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 20:40:55.131729 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.131391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33558)
0806 20:40:55.131526 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.131597 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 20:40:55.142333 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5574
I20260806 20:40:55.163475 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5723
I20260806 20:40:55.177379 5886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.164235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.23.200.3:34775)
0806 20:40:55.164442 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:55.164464 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:55.164537 (+ 73us) server_negotiation.cc:407] Connection header received
0806 20:40:55.164766 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:55.164771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:55.164878 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:55.165073 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:55.166599 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:55.167313 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:55.168626 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:55.169022 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:55.169158 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:55.171883 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:55.171894 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:55.171908 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:55.171940 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:55.174558 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:55.175273 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:55.175280 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:55.175282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:55.175416 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:55.175797 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:55.175803 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:55.175806 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:55.176159 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:55.176471 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:55.176787 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 20:40:55.177026 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:40:55.178025 5852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:55.212790 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.212297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33566)
0806 20:40:55.212525 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.212612 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 20:40:55.213877 5922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:55.213462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45195 (local address 127.0.0.1:33582)
0806 20:40:55.213675 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:55.213743 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 20:40:55.269912 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5723
I20260806 20:40:55.287985 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5334
I20260806 20:40:55.289186 5398 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:40:55.402398 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5334
I20260806 20:40:55.428936 24352 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 20:40:55.774149 24352 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 20:40:55.774387 24352 mini_ranger.cc:66] Stopping Ranger...
I20260806 20:40:56.119803 24352 mini_ranger.cc:68] Stopped Ranger
2026-08-06 20:40:56.120 UTC [4354] LOG: received smart shutdown request
2026-08-06 20:40:56.123 UTC [4354] LOG: background worker "logical replication launcher" (PID 4363) exited with exit code 1
2026-08-06 20:40:56.123 UTC [4358] LOG: shutting down
2026-08-06 20:40:56.267 UTC [4354] LOG: database system is shut down
2026-08-06T20:40:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52818 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:40:56 dist-test-slave-tm1g krb5kdc[5935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:40:56 dist-test-slave-tm1g krb5kdc[5935](info): set up 2 sockets
Aug 06 20:40:56 dist-test-slave-tm1g krb5kdc[5935](info): commencing operation
krb5kdc: starting...
W20260806 20:40:57.510007 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:40:57 dist-test-slave-tm1g krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:40:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:40:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:40:57.737774 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44489
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:44489,127.23.200.61:41203,127.23.200.60:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:40:58.077764 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:58.078032 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:58.078121 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:58.085196 5948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:40:58.085312 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:58.085358 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:58.085417 5948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:40:58.085466 5948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:40:58.094856 5948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:44489,127.23.200.61:41203,127.23.200.60:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:58.096082 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:58.097339 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:58.109174 5954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:58.109125 5955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:58.109220 5957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:58.110095 5948 server_base.cc:1034] running on GCE node
I20260806 20:40:58.110883 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:58.112182 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:58.113389 5948 hybrid_clock.cc:648] HybridClock initialized: now 1786048858113347 us; error 32 us; skew 500 ppm
I20260806 20:40:58.113796 5948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:58 dist-test-slave-tm1g krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:58.121078 5948 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:40:58.122597 5948 webserver.cc:460] Webserver started at http://127.23.200.62:35663/ using document root <none> and password file <none>
I20260806 20:40:58.123283 5948 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:58.123443 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:58.123771 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:40:58.126786 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "d3bdb2ec9ddf4443a723c827d0a14a50"
format_stamp: "Formatted at 2026-08-06 20:40:58 on dist-test-slave-tm1g"
server_key: "762d8048027bfeeaee4ff21081cb6868"
server_key_iv: "10bc83ab9bc887f2273b1f34c1bc0982"
server_key_version: "encryptionkey@0"
I20260806 20:40:58.127544 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "d3bdb2ec9ddf4443a723c827d0a14a50"
format_stamp: "Formatted at 2026-08-06 20:40:58 on dist-test-slave-tm1g"
server_key: "762d8048027bfeeaee4ff21081cb6868"
server_key_iv: "10bc83ab9bc887f2273b1f34c1bc0982"
server_key_version: "encryptionkey@0"
I20260806 20:40:58.132596 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:40:58.136049 5964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:58.137338 5948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:40:58.137548 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "d3bdb2ec9ddf4443a723c827d0a14a50"
format_stamp: "Formatted at 2026-08-06 20:40:58 on dist-test-slave-tm1g"
server_key: "762d8048027bfeeaee4ff21081cb6868"
server_key_iv: "10bc83ab9bc887f2273b1f34c1bc0982"
server_key_version: "encryptionkey@0"
I20260806 20:40:58.137753 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:58.153246 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:58.157541 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:58.157835 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:58.177918 5948 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:44489
I20260806 20:40:58.177948 6016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:44489 every 8 connection(s)
I20260806 20:40:58.179394 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:40:58.183099 6017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:58.185209 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 5948
I20260806 20:40:58.185487 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:40:58.186359 24352 external_mini_cluster.cc:1428] Setting key 5c07aa622851d4c0c465d83aabe14242
I20260806 20:40:58.192626 6017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:40:58 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.61@KRBTEST.COM" created.
I20260806 20:40:58.215727 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.196569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.23.200.62:44489)
0806 20:40:58.198873 (+ 2304us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:58.198881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:58.198930 (+ 49us) server_negotiation.cc:407] Connection header received
0806 20:40:58.200001 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:58.200007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:58.200223 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:58.200429 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:58.201692 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.202973 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.204352 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.204903 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.205117 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.208731 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:58.208739 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:58.208749 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:58.208776 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:58.211615 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.212307 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.212321 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.212324 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.212464 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.212880 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.212890 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.212893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.213378 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:58.213546 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:58.214239 (+ 693us) server_negotiation.cc:299] Negotiation successful
0806 20:40:58.214665 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":2081,"spinlock_wait_cycles":16128,"thread_start_us":1810,"threads_started":1}
I20260806 20:40:58.215754 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.196569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44489 (local address 127.0.0.1:36960)
0806 20:40:58.197486 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.197531 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:58.199643 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:58.200439 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:58.200445 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:58.201163 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:58.201499 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.201515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.203127 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.203134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.204211 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.204223 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.205141 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.205148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.205349 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.206019 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:58.206040 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:58.208510 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:58.211746 (+ 3236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.211759 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.211771 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.212174 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.212577 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.212583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.212586 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.212738 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.213539 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:58.213550 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:58.213836 (+ 286us) client_negotiation.cc:769] Sending connection context
0806 20:40:58.214143 (+ 307us) client_negotiation.cc:240] Negotiation successful
0806 20:40:58.214860 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":112,"thread_start_us":437,"threads_started":1}
I20260806 20:40:58.218909 6017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:40:58.219990 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.219490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:55920)
0806 20:40:58.219692 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.219829 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 20:40:58.220263 5967 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
I20260806 20:40:58.221828 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.221512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:55922)
0806 20:40:58.221646 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.221717 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 20:40:58.222162 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41203: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab.
Entry for principal kudu/127.23.200.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab.
I20260806 20:40:58.241708 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:41203
--webserver_interface=127.23.200.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab
--principal=kudu/127.23.200.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:44489,127.23.200.61:41203,127.23.200.60:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:40:58.271581 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } attempt: 1
I20260806 20:40:58.272825 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.272372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:55934)
0806 20:40:58.272569 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.272677 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 20:40:58.273823 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.273470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:55940)
0806 20:40:58.273625 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.273690 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:40:58.274013 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41203: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
I20260806 20:40:58.342347 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } attempt: 2
I20260806 20:40:58.343582 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.343062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:55944)
0806 20:40:58.343329 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.343433 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 20:40:58.344771 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.344336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:55958)
0806 20:40:58.344528 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.344623 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260806 20:40:58.345068 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41203: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
I20260806 20:40:58.436344 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } attempt: 3
I20260806 20:40:58.437603 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.437146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38118)
0806 20:40:58.437349 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.437470 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 20:40:58.438638 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.438267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38124)
0806 20:40:58.438449 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.438516 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260806 20:40:58.438906 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41203: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
W20260806 20:40:58.568069 6025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:58.568396 6025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:58.568572 6025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:58.575435 6025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:40:58.575536 6025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:58.575649 6025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:58.575758 6025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:40:58.575825 6025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:40:58.582149 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } attempt: 4
I20260806 20:40:58.583443 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.582906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38138)
0806 20:40:58.583218 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.583310 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260806 20:40:58.584503 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.584092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38150)
0806 20:40:58.584279 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.584379 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 20:40:58.584710 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41203: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41203: connect: Connection refused (error 111)
I20260806 20:40:58.585919 6025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:44489,127.23.200.61:41203,127.23.200.60:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab
--principal=kudu/127.23.200.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:41203
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.200.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:58.587260 6025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:58.588496 6025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:40:58.600054 6030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:58.599977 6033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:58.600356 6031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:58.601215 6025 server_base.cc:1034] running on GCE node
I20260806 20:40:58.601960 6025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:58.603384 6025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:58.604617 6025 hybrid_clock.cc:648] HybridClock initialized: now 1786048858604566 us; error 40 us; skew 500 ppm
I20260806 20:40:58.604950 6025 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:58 dist-test-slave-tm1g krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:58.607992 6025 init.cc:377] Logged in from keytab as kudu/127.23.200.61@KRBTEST.COM (short username kudu)
I20260806 20:40:58.609308 6025 webserver.cc:460] Webserver started at http://127.23.200.61:44201/ using document root <none> and password file <none>
I20260806 20:40:58.609925 6025 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:58.610021 6025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:58.610253 6025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:40:58.613204 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data/instance:
uuid: "f069307d6a834f18812d579f38f83181"
format_stamp: "Formatted at 2026-08-06 20:40:58 on dist-test-slave-tm1g"
server_key: "dfd582fb3449021684f3e0dc2a2cc635"
server_key_iv: "a5f0dc33799fd99d2bafb2fd8e3c7bbd"
server_key_version: "encryptionkey@0"
I20260806 20:40:58.613933 6025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal/instance:
uuid: "f069307d6a834f18812d579f38f83181"
format_stamp: "Formatted at 2026-08-06 20:40:58 on dist-test-slave-tm1g"
server_key: "dfd582fb3449021684f3e0dc2a2cc635"
server_key_iv: "a5f0dc33799fd99d2bafb2fd8e3c7bbd"
server_key_version: "encryptionkey@0"
I20260806 20:40:58.618839 6025 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 20:40:58.621914 6040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:58.623207 6025 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:40:58.623342 6025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal
uuid: "f069307d6a834f18812d579f38f83181"
format_stamp: "Formatted at 2026-08-06 20:40:58 on dist-test-slave-tm1g"
server_key: "dfd582fb3449021684f3e0dc2a2cc635"
server_key_iv: "a5f0dc33799fd99d2bafb2fd8e3c7bbd"
server_key_version: "encryptionkey@0"
I20260806 20:40:58.623507 6025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:58.647969 6025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:58.651770 6025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:58.651998 6025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:58.670372 6025 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.61:41203
I20260806 20:40:58.670472 6093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.61:41203 every 8 connection(s)
I20260806 20:40:58.671670 6025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
I20260806 20:40:58.673341 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6025
I20260806 20:40:58.673566 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-1/wal/instance
I20260806 20:40:58.674052 24352 external_mini_cluster.cc:1428] Setting key f5ffa8d11e63283caed9caf60006ec1f
I20260806 20:40:58.675303 6094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:58.681497 6094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:40:58 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.60@KRBTEST.COM" created.
I20260806 20:40:58.702840 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.684354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.23.200.62:44489)
0806 20:40:58.684594 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:58.684603 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:58.685695 (+ 1092us) server_negotiation.cc:407] Connection header received
0806 20:40:58.687566 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:58.687575 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:58.687666 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:58.687788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:58.689206 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.690373 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.691860 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.692633 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.692794 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.696676 (+ 3882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:58.696681 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:58.696684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:58.696716 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:58.699644 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.700532 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.700536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.700538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.700655 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.701135 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.701138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.701140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.701395 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:58.701512 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:58.702443 (+ 931us) server_negotiation.cc:299] Negotiation successful
0806 20:40:58.702610 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:40:58.704301 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.684733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44489 (local address 127.0.0.1:47458)
0806 20:40:58.685521 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.685566 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:58.687308 (+ 1742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:58.688004 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:58.688012 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:58.688508 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:58.689036 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.689053 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.690521 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.690524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.691720 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.691732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.692842 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.692847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.693039 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.693738 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:58.693763 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:58.696470 (+ 2707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:58.699771 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.699781 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.699798 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.700416 (+ 618us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.700775 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.700783 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.700789 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.701028 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.701531 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:58.701541 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:58.701993 (+ 452us) client_negotiation.cc:769] Sending connection context
0806 20:40:58.702332 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 20:40:58.702873 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":332,"threads_started":1}
I20260806 20:40:58.706087 6094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:40:58 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:40:58.720224 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.706784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38164)
0806 20:40:58.707035 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.707053 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:58.707151 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:58.708223 (+ 1072us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:58.708227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:58.708643 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:58.708760 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.708767 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.710066 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.710073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.710949 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.710960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.711702 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.711708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.711964 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.712498 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:58.712512 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:58.714704 (+ 2192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:58.717782 (+ 3078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.717787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.717791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.718142 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.718543 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.718549 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.718552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.718704 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.719283 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:58.719288 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:58.719426 (+ 138us) client_negotiation.cc:769] Sending connection context
0806 20:40:58.719581 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 20:40:58.719853 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":41}
I20260806 20:40:58.720337 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.707106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.23.200.61:41203)
0806 20:40:58.707832 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:58.707838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:58.707862 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:40:58.707901 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:58.707906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:58.708075 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:58.708230 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:58.708915 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.709932 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.711076 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.711567 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.711696 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.714916 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:58.714927 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:58.714938 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:58.714964 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:58.717649 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.718268 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.718283 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.718285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.718438 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.718807 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.718813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.718816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.719150 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:58.719344 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:58.719723 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 20:40:58.719970 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":585,"thread_start_us":446,"threads_started":1}
I20260806 20:40:58.722625 6094 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:40:58.723662 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.723195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44904)
0806 20:40:58.723408 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.723510 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 20:40:58.723922 6042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:58.725384 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.725064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44918)
0806 20:40:58.725191 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.725264 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:40:58.725672 6094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab.
Entry for principal kudu/127.23.200.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab.
I20260806 20:40:58.729633 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:38985
--webserver_interface=127.23.200.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab
--principal=kudu/127.23.200.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:41583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:44489,127.23.200.61:41203,127.23.200.60:38985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:40:58.774973 6094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 1
I20260806 20:40:58.776175 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.775729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44934)
0806 20:40:58.775926 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.776027 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 20:40:58.777206 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.776901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44948)
0806 20:40:58.777050 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.777111 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:40:58.777458 6094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:58.845696 6094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 2
I20260806 20:40:58.846913 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.846464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44950)
0806 20:40:58.846660 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.846748 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 20:40:58.847892 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.847579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44956)
0806 20:40:58.847711 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.847774 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 20:40:58.848130 6094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:58.884004 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } attempt: 5
Aug 06 20:40:58 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:40:58.897874 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.885031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38166 (local address 127.23.200.61:41203)
0806 20:40:58.885268 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:58.885276 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:58.885315 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:40:58.885372 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:58.885375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:58.885434 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:58.885547 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:58.886396 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.887101 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.888297 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.888768 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.888924 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.892582 (+ 3658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:58.892588 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:58.892591 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:58.892623 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:58.895292 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.896086 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.896089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.896092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.896209 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:58.896634 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:58.896637 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:58.896639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:58.896928 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:58.897042 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:58.897476 (+ 434us) server_negotiation.cc:299] Negotiation successful
0806 20:40:58.897639 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 20:40:58.898038 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.884796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38166)
0806 20:40:58.885002 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.885026 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:58.885178 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:58.885590 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:58.885597 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:58.886007 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:58.886197 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.886206 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.887241 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.887247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.888143 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:58.888155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:58.888937 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:58.888942 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:58.889138 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:58.889788 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:58.889807 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:58.892385 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:58.895456 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.895462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.895466 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.895966 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.896315 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:58.896320 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:58.896324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:58.896521 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:58.897049 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:58.897054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:58.897228 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 20:40:58.897384 (+ 156us) client_negotiation.cc:240] Negotiation successful
0806 20:40:58.897688 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 20:40:58.899274 6017 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:40:58.900442 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.899925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44972)
0806 20:40:58.900129 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.900239 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 20:40:58.901465 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.901027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44986)
0806 20:40:58.901192 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.901311 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 20:40:58.901670 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:58.939395 6094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 3
I20260806 20:40:58.940675 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.940199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:44994)
0806 20:40:58.940400 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.940491 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 20:40:58.941867 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.941438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45006)
0806 20:40:58.941688 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.941755 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260806 20:40:58.942103 6094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:58.952929 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 1
I20260806 20:40:58.954092 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.953560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45022)
0806 20:40:58.953773 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.953884 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 20:40:58.955149 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:58.954740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45030)
0806 20:40:58.954919 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:58.955003 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 20:40:58.955390 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:59.023650 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 2
I20260806 20:40:59.024950 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.024378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45032)
0806 20:40:59.024628 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.024734 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 20:40:59.026091 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.025644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45044)
0806 20:40:59.025821 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.025929 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 20:40:59.026371 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
W20260806 20:40:59.064079 6101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:40:59.064333 6101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:40:59.064483 6101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:40:59.071609 6101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:40:59.071703 6101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:40:59.071805 6101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:40:59.071908 6101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:40:59.071991 6101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:40:59.081314 6101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:41583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:44489,127.23.200.61:41203,127.23.200.60:38985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab
--principal=kudu/127.23.200.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:38985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.200.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:40:59.082659 6101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:40:59.083935 6101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 20:40:59.085331 6094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 4
I20260806 20:40:59.086583 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.086049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45054)
0806 20:40:59.086266 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.086412 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 20:40:59.087601 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.087273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45064)
0806 20:40:59.087424 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.087502 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 20:40:59.087848 6094 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
W20260806 20:40:59.095924 6109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:59.095968 6111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:40:59.096199 6107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:40:59.097102 6101 server_base.cc:1034] running on GCE node
I20260806 20:40:59.097791 6101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:40:59.099123 6101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:40:59.100350 6101 hybrid_clock.cc:648] HybridClock initialized: now 1786048859100318 us; error 41 us; skew 500 ppm
I20260806 20:40:59.100683 6101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:40:59.103840 6101 init.cc:377] Logged in from keytab as kudu/127.23.200.60@KRBTEST.COM (short username kudu)
I20260806 20:40:59.105316 6101 webserver.cc:460] Webserver started at http://127.23.200.60:44551/ using document root <none> and password file <none>
I20260806 20:40:59.105986 6101 fs_manager.cc:359] Metadata directory not provided
I20260806 20:40:59.106132 6101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:40:59.106495 6101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:40:59.109352 6101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data/instance:
uuid: "9999302bd5e64cc58c3019e5795bab4c"
format_stamp: "Formatted at 2026-08-06 20:40:59 on dist-test-slave-tm1g"
server_key: "8fd59c557929274e393bc4700e4c3fa8"
server_key_iv: "f7eb6d9ab5d1653d4746ce023f6b7108"
server_key_version: "encryptionkey@0"
I20260806 20:40:59.110241 6101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal/instance:
uuid: "9999302bd5e64cc58c3019e5795bab4c"
format_stamp: "Formatted at 2026-08-06 20:40:59 on dist-test-slave-tm1g"
server_key: "8fd59c557929274e393bc4700e4c3fa8"
server_key_iv: "f7eb6d9ab5d1653d4746ce023f6b7108"
server_key_version: "encryptionkey@0"
I20260806 20:40:59.115172 6101 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 20:40:59.118410 6118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:40:59.119714 6101 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:40:59.119869 6101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal
uuid: "9999302bd5e64cc58c3019e5795bab4c"
format_stamp: "Formatted at 2026-08-06 20:40:59 on dist-test-slave-tm1g"
server_key: "8fd59c557929274e393bc4700e4c3fa8"
server_key_iv: "f7eb6d9ab5d1653d4746ce023f6b7108"
server_key_version: "encryptionkey@0"
I20260806 20:40:59.120069 6101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:40:59.132632 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 3
I20260806 20:40:59.133679 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.133246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45072)
0806 20:40:59.133437 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.133521 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 20:40:59.134733 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.134291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45086)
0806 20:40:59.134564 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.134629 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 20:40:59.134946 6017 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:38985: Network error: Client connection negotiation failed: client connection to 127.23.200.60:38985: connect: Connection refused (error 111)
I20260806 20:40:59.147806 6101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:40:59.152171 6101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:40:59.152459 6101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:40:59.171099 6101 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.60:38985
I20260806 20:40:59.171099 6169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.60:38985 every 8 connection(s)
I20260806 20:40:59.172377 6101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
I20260806 20:40:59.175822 6170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:40:59.180864 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6101
I20260806 20:40:59.181092 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786048651485794-24352-0/minicluster-data/master-2/wal/instance
I20260806 20:40:59.181651 24352 external_mini_cluster.cc:1428] Setting key a5ffb67f53030d641311ee5a24661582
I20260806 20:40:59.183090 6170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
I20260806 20:40:59.205224 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.187968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.23.200.62:44489)
0806 20:40:59.188236 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.188243 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.188789 (+ 546us) server_negotiation.cc:407] Connection header received
0806 20:40:59.190207 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.190210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.190271 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.190399 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.191773 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.192589 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.194368 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.195060 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.195202 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.198742 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.198746 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.198750 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.198781 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.202143 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.202927 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.202933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.202939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.203106 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.203534 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.203541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.203544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.203801 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.203967 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.204618 (+ 651us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.204864 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 20:40:59.206377 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.183592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.23.200.60:38985)
0806 20:40:59.187562 (+ 3970us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.187577 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.187864 (+ 287us) server_negotiation.cc:407] Connection header received
0806 20:40:59.189955 (+ 2091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.189981 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.190259 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.190523 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.191922 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.193475 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.195107 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.195693 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.195841 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.200001 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.200013 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.200028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.200071 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.203328 (+ 3257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.204000 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.204008 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.204012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.204142 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.204529 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.204533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.204537 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.204925 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.205219 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.205686 (+ 467us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.205921 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":18,"server-negotiator.queue_time_us":3735,"thread_start_us":3496,"threads_started":1}
I20260806 20:40:59.206416 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.187932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:44489 (local address 127.0.0.1:47468)
0806 20:40:59.188619 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.188666 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:59.189955 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:59.190486 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:59.190493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:59.191223 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:59.191601 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.191618 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.192759 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.192765 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.194211 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.194226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.195293 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.195301 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.195577 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.196315 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:59.196353 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:59.198560 (+ 2207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:59.202269 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.202282 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.202290 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.202804 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.203226 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.203229 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.203231 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.203402 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.203934 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:59.203942 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:59.204244 (+ 302us) client_negotiation.cc:769] Sending connection context
0806 20:40:59.204523 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 20:40:59.205020 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"mutex_wait_us":24,"thread_start_us":272,"threads_started":1}
I20260806 20:40:59.209592 6170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:40:59.222261 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.210193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41203 (local address 127.0.0.1:38178)
0806 20:40:59.210373 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.210395 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:59.210464 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:59.210905 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:59.210910 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:59.211266 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:59.211395 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.211403 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.212366 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.212372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.213218 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.213230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.213938 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.213943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.214146 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.214710 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:59.214736 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:59.216610 (+ 1874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:59.219961 (+ 3351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.219966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.219970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.220353 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.220730 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.220735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.220739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.220886 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.221369 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:59.221373 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:59.221522 (+ 149us) client_negotiation.cc:769] Sending connection context
0806 20:40:59.221659 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 20:40:59.221929 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260806 20:40:59.222352 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.210499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.23.200.61:41203)
0806 20:40:59.210652 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.210656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.210669 (+ 13us) server_negotiation.cc:407] Connection header received
0806 20:40:59.210714 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.210718 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.210769 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.210872 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.211556 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.212232 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.213367 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.213788 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.213927 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.216796 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.216801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.216804 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.216836 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.219805 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.220482 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.220489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.220492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.220619 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.220996 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.221003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.221006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.221250 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.221408 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.221762 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.221970 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 20:40:59.223301 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.209634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47472 (local address 127.23.200.62:44489)
0806 20:40:59.209812 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.209817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.209834 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:40:59.209943 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.209949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.210019 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.210150 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.211422 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.212170 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.213394 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.213817 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.213948 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.218169 (+ 4221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.218174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.218177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.218205 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.220734 (+ 2529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.221379 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.221386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.221389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.221519 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.221894 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.221899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.221902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.222171 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.222690 (+ 519us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.222806 (+ 116us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.223024 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:40:59.223526 6170 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048859, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048857, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:40:59.237675 6174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.224456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45114)
0806 20:40:59.224701 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.224718 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:59.224818 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:59.225210 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:59.225213 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:59.225532 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:59.225647 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.225654 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.228003 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.228006 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.228836 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.228847 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.229591 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.229598 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.229807 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.230379 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:59.230406 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:59.232203 (+ 1797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:59.235295 (+ 3092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.235301 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.235310 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.235818 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.236193 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.236198 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.236202 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.236360 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.236824 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:59.236830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:59.237002 (+ 172us) client_negotiation.cc:769] Sending connection context
0806 20:40:59.237123 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 20:40:59.237315 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260806 20:40:59.237876 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.224367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45114 (local address 127.23.200.60:38985)
0806 20:40:59.224543 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.224548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.224770 (+ 222us) server_negotiation.cc:407] Connection header received
0806 20:40:59.224949 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.224953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.225013 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.225094 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.225781 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.226882 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.228981 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.229446 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.229537 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.232401 (+ 2864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.232407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.232412 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.232453 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.235150 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.235954 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.235958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.235960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.236076 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.236467 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.236470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.236472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.236712 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.236813 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.237254 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.237512 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 20:40:59.239848 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.224987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38180 (local address 127.23.200.61:41203)
0806 20:40:59.225213 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.225218 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.225235 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:40:59.225334 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.225339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.225404 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.225501 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.227014 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.227846 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.229082 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.229491 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.229633 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.233841 (+ 4208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.233845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.233849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.233877 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.236803 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.237541 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.237546 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.237549 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.237715 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.238397 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.238401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.238414 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.238664 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.238953 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.239286 (+ 333us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.239504 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 20:40:59.243508 6170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: Bootstrap starting.
I20260806 20:40:59.244613 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:40:59.246809 6170 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:59.247798 6170 log.cc:824] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:59.251113 6170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: No bootstrap required, opened a new log
I20260806 20:40:59.264274 6170 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.264747 6170 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:59.264859 6170 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9999302bd5e64cc58c3019e5795bab4c, State: Initialized, Role: FOLLOWER
I20260806 20:40:59.265676 6170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.272544 6021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.249537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47488 (local address 127.23.200.62:44489)
0806 20:40:59.249758 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.249766 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.250386 (+ 620us) server_negotiation.cc:407] Connection header received
0806 20:40:59.250667 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.250673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.250757 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.250837 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.253724 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.255156 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.258890 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.259539 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.259639 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.263639 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.263644 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.263648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.263688 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.267891 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.269320 (+ 1429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.269329 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.269335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.269538 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.270334 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.270343 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.270348 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.270806 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.271120 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.271796 (+ 676us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.272116 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 20:40:59.272742 6183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.273169 6183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.275127 6170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:40:59.277110 6099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.250542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.23.200.61:41203)
0806 20:40:59.250701 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.250706 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.258963 (+ 8257us) server_negotiation.cc:407] Connection header received
0806 20:40:59.259206 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.259213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.259288 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.259355 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.261792 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.262915 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.265255 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.265928 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.266045 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.270372 (+ 4327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.270379 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.270383 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.270420 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.274053 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.274934 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.274940 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.274945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.275118 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.275619 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.275624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.275627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.275932 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.276195 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.276512 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.276738 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:40:59.278080 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.250915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.23.200.60:38985)
0806 20:40:59.253932 (+ 3017us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.253937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.258653 (+ 4716us) server_negotiation.cc:407] Connection header received
0806 20:40:59.258926 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.258938 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.259014 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.259094 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.261792 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.262943 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.265231 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.265970 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.266052 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.270405 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.270411 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.270415 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.270448 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.274562 (+ 4114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.275540 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.275545 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.275549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.275831 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.276293 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.276298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.276302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.276693 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.276943 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.277509 (+ 566us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.277725 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2880}
W20260806 20:40:59.288755 6194 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:40:59.288887 6194 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:40:59.284451 6101 master_runner.cc:186] Error getting master registration for 127.23.200.62:44489: OK, instance_id { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" instance_seqno: 1786048858172211 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 20:40:59.294143 6101 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 20:40:59.312180 6017 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 4
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
I20260806 20:40:59.327438 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.312998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.23.200.60:38985)
0806 20:40:59.313190 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.313197 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.313217 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:40:59.313269 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.313275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.313355 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.313462 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.314260 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.315498 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.317364 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.317814 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.317910 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.321512 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.321517 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.321520 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.321552 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.324532 (+ 2980us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.325197 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.325203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.325215 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.325394 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.325803 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.325808 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.325815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.326100 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.326267 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.326786 (+ 519us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.327037 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:40:59.328553 6020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.312746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45130)
0806 20:40:59.312962 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.312997 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:59.313121 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:59.313499 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:59.313503 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:59.313938 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:59.314118 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.314127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.315676 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.315682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.317197 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.317213 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.317973 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.317979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.318191 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.318838 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:59.318862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:59.321323 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:59.324664 (+ 3341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.324671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.324675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.325069 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.325538 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.325544 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.325547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.325692 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.326237 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:59.326242 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:59.326423 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 20:40:59.326667 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 20:40:59.328169 (+ 1502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 20:40:59.334846 6017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: Bootstrap starting.
I20260806 20:40:59.339052 6017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:59.340395 6017 log.cc:824] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:59.343856 6017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: No bootstrap required, opened a new log
I20260806 20:40:59.354753 6017 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.355129 6017 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:59.355243 6017 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bdb2ec9ddf4443a723c827d0a14a50, State: Initialized, Role: FOLLOWER
I20260806 20:40:59.355865 6017 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.356779 6198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.357134 6198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.358072 6017 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 20:40:59.368927 6210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:40:59.369146 6210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:40:59.368141 5948 master_runner.cc:186] Error getting master registration for 127.23.200.61:41203: OK, instance_id { permanent_uuid: "f069307d6a834f18812d579f38f83181" instance_seqno: 1786048858665078 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 20:40:59.379369 5948 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 20:40:59.387076 6094 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } attempt: 5
Aug 06 20:40:59 dist-test-slave-tm1g krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048858, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
W20260806 20:40:59.404498 6124 tablet.cc:2369] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:40:59.405360 6097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.387695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:38985 (local address 127.0.0.1:45144)
0806 20:40:59.387960 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 20:40:59.387981 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:40:59.388088 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:40:59.388494 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:40:59.388499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:40:59.389044 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:40:59.389246 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.389258 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.390788 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.390794 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.392264 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:40:59.392280 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.393562 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.393569 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:40:59.393791 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.395263 (+ 1472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:40:59.395290 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:40:59.398403 (+ 3113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:40:59.402556 (+ 4153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.402562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.402566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.403105 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.403521 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:40:59.403526 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:40:59.403530 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:40:59.403698 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:40:59.404309 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:40:59.404314 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:40:59.404471 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 20:40:59.404637 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 20:40:59.404947 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 20:40:59.406450 6173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:40:59.387802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45144 (local address 127.23.200.60:38985)
0806 20:40:59.387999 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 20:40:59.388005 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:40:59.388031 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:40:59.388228 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:40:59.388234 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:40:59.388322 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:40:59.388436 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:40:59.389423 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.390629 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.392430 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:40:59.393204 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:40:59.393334 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:40:59.398602 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:40:59.398614 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:40:59.398619 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:40:59.398668 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:40:59.402417 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.403231 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.403236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.403240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.403395 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:40:59.403814 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:40:59.403820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:40:59.403824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:40:59.404176 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:40:59.404339 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 20:40:59.405756 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0806 20:40:59.406023 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 20:40:59.410610 5970 tablet.cc:2369] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:40:59.412703 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: Bootstrap starting.
I20260806 20:40:59.416795 6094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: Neither blocks nor log segments found. Creating new log.
I20260806 20:40:59.418100 6094 log.cc:824] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: Log is configured to *not* fsync() on all Append() calls
I20260806 20:40:59.421404 6094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: No bootstrap required, opened a new log
I20260806 20:40:59.432528 6094 raft_consensus.cc:348] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.432914 6094 raft_consensus.cc:374] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:40:59.433036 6094 raft_consensus.cc:729] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f069307d6a834f18812d579f38f83181, State: Initialized, Role: FOLLOWER
I20260806 20:40:59.433669 6094 consensus_queue.cc:260] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.434652 6213 sys_catalog.cc:455] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.435143 6213 sys_catalog.cc:458] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.436331 6094 sys_catalog.cc:564] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:40:59.459797 6025 master_runner.cc:263] No leader master found from master f069307d6a834f18812d579f38f83181
I20260806 20:40:59.459947 6025 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 20:40:59.462692 6224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:40:59.462909 6224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:40:59.627260 6213 raft_consensus.cc:482] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:40:59.627629 6213 raft_consensus.cc:504] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.629498 6213 leader_election.cc:288] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3bdb2ec9ddf4443a723c827d0a14a50 (127.23.200.62:44489), 9999302bd5e64cc58c3019e5795bab4c (127.23.200.60:38985)
I20260806 20:40:59.630863 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f069307d6a834f18812d579f38f83181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" is_pre_election: true
I20260806 20:40:59.631494 5991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f069307d6a834f18812d579f38f83181 in term 0.
I20260806 20:40:59.632368 6043 leader_election.cc:302] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3bdb2ec9ddf4443a723c827d0a14a50, f069307d6a834f18812d579f38f83181; no voters:
I20260806 20:40:59.632903 6213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:40:59.633061 6213 raft_consensus.cc:482] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:40:59.633178 6213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:59.635236 6145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f069307d6a834f18812d579f38f83181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9999302bd5e64cc58c3019e5795bab4c" is_pre_election: true
I20260806 20:40:59.635929 6145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f069307d6a834f18812d579f38f83181 in term 0.
I20260806 20:40:59.640061 6213 raft_consensus.cc:504] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.641556 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f069307d6a834f18812d579f38f83181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50"
I20260806 20:40:59.641784 5991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:59.645455 5991 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f069307d6a834f18812d579f38f83181 in term 1.
I20260806 20:40:59.645970 6043 leader_election.cc:302] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3bdb2ec9ddf4443a723c827d0a14a50, f069307d6a834f18812d579f38f83181; no voters:
I20260806 20:40:59.649953 6145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f069307d6a834f18812d579f38f83181" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9999302bd5e64cc58c3019e5795bab4c"
I20260806 20:40:59.650202 6145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:40:59.653692 6145 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f069307d6a834f18812d579f38f83181 in term 1.
W20260806 20:40:59.654429 6046 tablet.cc:2369] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:40:59.658916 6213 leader_election.cc:288] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [CANDIDATE]: Term 1 election: Requested vote from peers d3bdb2ec9ddf4443a723c827d0a14a50 (127.23.200.62:44489), 9999302bd5e64cc58c3019e5795bab4c (127.23.200.60:38985)
I20260806 20:40:59.659300 6213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:40:59.659843 6213 raft_consensus.cc:686] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [term 1 LEADER]: Becoming Leader. State: Replica: f069307d6a834f18812d579f38f83181, State: Running, Role: LEADER
I20260806 20:40:59.661392 6213 consensus_queue.cc:237] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } }
I20260806 20:40:59.671625 6226 sys_catalog.cc:455] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f069307d6a834f18812d579f38f83181. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.672129 6226 sys_catalog.cc:458] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:59.677002 6228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:40:59.679337 6228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:40:59.698717 6145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [term 1 FOLLOWER]: Refusing update from remote peer f069307d6a834f18812d579f38f83181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:40:59.699919 6226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:40:59.708053 5991 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [term 1 FOLLOWER]: Refusing update from remote peer f069307d6a834f18812d579f38f83181: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:40:59.708923 6226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:40:59.738541 6198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f069307d6a834f18812d579f38f83181. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.738919 6198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.746354 6183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f069307d6a834f18812d579f38f83181. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.746640 6183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.753286 6229 mvcc.cc:205] Tried to move back new op lower bound from 7315656129313796096 to 7315656129188446208. Current Snapshot: MvccSnapshot[applied={T|T < 7315656129313796096}]
I20260806 20:40:59.756743 6230 mvcc.cc:205] Tried to move back new op lower bound from 7315656129313796096 to 7315656129188446208. Current Snapshot: MvccSnapshot[applied={T|T < 7315656129313796096}]
I20260806 20:40:59.758822 6213 sys_catalog.cc:455] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.759292 6213 sys_catalog.cc:458] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:59.759891 6213 sys_catalog.cc:455] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.760255 6213 sys_catalog.cc:458] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181 [sys.catalog]: This master's current role is: LEADER
I20260806 20:40:59.761238 6228 catalog_manager.cc:1347] Generated new cluster ID: 1cc32208c2b241e8be7f819b8302c587
I20260806 20:40:59.761353 6228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:40:59.766861 6198 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.767117 6198 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.767649 6183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f069307d6a834f18812d579f38f83181" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bdb2ec9ddf4443a723c827d0a14a50" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 44489 } } peers { permanent_uuid: "f069307d6a834f18812d579f38f83181" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41203 } } peers { permanent_uuid: "9999302bd5e64cc58c3019e5795bab4c" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 38985 } } }
I20260806 20:40:59.767912 6183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:40:59.821496 6228 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:40:59.823661 6228 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:40:59.848362 6228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f069307d6a834f18812d579f38f83181: Generated new TSK 0
I20260806 20:40:59.849507 6228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:41:00.290999 6194 catalog_manager.cc:1259] Loaded cluster ID: 1cc32208c2b241e8be7f819b8302c587
I20260806 20:41:00.291160 6194 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: loading cluster ID for follower catalog manager: success
I20260806 20:41:00.293668 6194 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: acquiring CA information for follower catalog manager: success
I20260806 20:41:00.295496 6194 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9999302bd5e64cc58c3019e5795bab4c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 20:41:00.371223 6210 catalog_manager.cc:1259] Loaded cluster ID: 1cc32208c2b241e8be7f819b8302c587
I20260806 20:41:00.371356 6210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: loading cluster ID for follower catalog manager: success
I20260806 20:41:00.373973 6210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: acquiring CA information for follower catalog manager: success
I20260806 20:41:00.375872 6210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d3bdb2ec9ddf4443a723c827d0a14a50: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 20:41:00.440831 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 5948
I20260806 20:41:00.442678 6011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:00.566200 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 5948
I20260806 20:41:00.593451 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6025
I20260806 20:41:00.594548 6088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:00.704931 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6025
I20260806 20:41:00.726531 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6101
I20260806 20:41:00.727707 6165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:00.837172 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6101
2026-08-06T20:41:00Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4439 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:41:00 dist-test-slave-tm1g krb5kdc[6248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:41:00 dist-test-slave-tm1g krb5kdc[6248](info): set up 2 sockets
Aug 06 20:41:00 dist-test-slave-tm1g krb5kdc[6248](info): commencing operation
krb5kdc: starting...
W20260806 20:41:01.965901 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:41:02 dist-test-slave-tm1g krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:41:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:41:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:41:02.195811 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36283
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:36283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:02.520130 6261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:02.520408 6261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:02.520498 6261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:02.527796 6261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:41:02.527896 6261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:02.527942 6261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:02.527997 6261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:41:02.528045 6261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:41:02.537497 6261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:36283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:02.538762 6261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:02.539970 6261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:02.551527 6269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:02.551604 6266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:02.551463 6267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:02.552443 6261 server_base.cc:1034] running on GCE node
I20260806 20:41:02.553108 6261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:02.554422 6261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:02.555611 6261 hybrid_clock.cc:648] HybridClock initialized: now 1786048862555560 us; error 38 us; skew 500 ppm
I20260806 20:41:02.555894 6261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:02 dist-test-slave-tm1g krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048862, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:02.558790 6261 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:41:02.560199 6261 webserver.cc:460] Webserver started at http://127.23.200.62:39123/ using document root <none> and password file <none>
I20260806 20:41:02.560873 6261 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:02.561021 6261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:02.561335 6261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:02.564289 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "065a967a6c4243c19bb33eff0699acd2"
format_stamp: "Formatted at 2026-08-06 20:41:02 on dist-test-slave-tm1g"
server_key: "67d1b3fc54eb7ca83a9ff617818e0d6c"
server_key_iv: "24431151599b34d31102c64979eb5fad"
server_key_version: "encryptionkey@0"
I20260806 20:41:02.565047 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "065a967a6c4243c19bb33eff0699acd2"
format_stamp: "Formatted at 2026-08-06 20:41:02 on dist-test-slave-tm1g"
server_key: "67d1b3fc54eb7ca83a9ff617818e0d6c"
server_key_iv: "24431151599b34d31102c64979eb5fad"
server_key_version: "encryptionkey@0"
I20260806 20:41:02.570178 6261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:41:02.573856 6276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:02.575098 6261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:02.575285 6261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "065a967a6c4243c19bb33eff0699acd2"
format_stamp: "Formatted at 2026-08-06 20:41:02 on dist-test-slave-tm1g"
server_key: "67d1b3fc54eb7ca83a9ff617818e0d6c"
server_key_iv: "24431151599b34d31102c64979eb5fad"
server_key_version: "encryptionkey@0"
I20260806 20:41:02.575484 6261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:02.589951 6261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:02.593669 6261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:02.593951 6261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:02.612419 6261 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:36283
I20260806 20:41:02.612437 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:36283 every 8 connection(s)
I20260806 20:41:02.613788 6261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:41:02.616717 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6261
I20260806 20:41:02.616961 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:41:02.617449 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:02.617699 24352 external_mini_cluster.cc:1428] Setting key 4dfb99d67ec1568210b5dc3daba42746
I20260806 20:41:02.631531 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2: Bootstrap starting.
Aug 06 20:41:02 dist-test-slave-tm1g krb5kdc[6248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048862, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:02.635239 6329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:02.636411 6329 log.cc:824] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:02.639694 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2: No bootstrap required, opened a new log
I20260806 20:41:02.641877 6332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:02.619465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33030 (local address 127.23.200.62:36283)
0806 20:41:02.620303 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:02.620324 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:02.620371 (+ 47us) server_negotiation.cc:407] Connection header received
0806 20:41:02.622367 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:02.622394 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:02.622854 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:02.623287 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:02.624681 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:02.626631 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:02.628211 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:02.629123 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:02.629349 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:02.634052 (+ 4703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:02.634060 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:02.634072 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:02.634112 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:02.637258 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:02.638124 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:02.638132 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:02.638139 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:02.638368 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:02.638758 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:02.638765 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:02.638769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:02.639464 (+ 695us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:02.639783 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:02.640152 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 20:41:02.640588 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":332,"threads_started":1}
I20260806 20:41:02.645699 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:41:02.648723 6329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065a967a6c4243c19bb33eff0699acd2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36283 } }
I20260806 20:41:02.649082 6329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:02.649201 6329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 065a967a6c4243c19bb33eff0699acd2, State: Initialized, Role: FOLLOWER
I20260806 20:41:02.649735 6329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065a967a6c4243c19bb33eff0699acd2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36283 } }
I20260806 20:41:02.649973 6329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:41:02.650133 6329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:41:02.650296 6329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:02.652370 6329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065a967a6c4243c19bb33eff0699acd2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36283 } }
I20260806 20:41:02.652848 6329 leader_election.cc:302] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 065a967a6c4243c19bb33eff0699acd2; no voters:
I20260806 20:41:02.653434 6329 leader_election.cc:288] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:41:02.653529 6334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:02.654250 6334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [term 1 LEADER]: Becoming Leader. State: Replica: 065a967a6c4243c19bb33eff0699acd2, State: Running, Role: LEADER
I20260806 20:41:02.654779 6334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065a967a6c4243c19bb33eff0699acd2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36283 } }
I20260806 20:41:02.655474 6329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:41:02.657783 6336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 065a967a6c4243c19bb33eff0699acd2. Latest consensus state: current_term: 1 leader_uuid: "065a967a6c4243c19bb33eff0699acd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065a967a6c4243c19bb33eff0699acd2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36283 } } }
I20260806 20:41:02.658198 6336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:02.658481 6335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "065a967a6c4243c19bb33eff0699acd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "065a967a6c4243c19bb33eff0699acd2" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36283 } } }
I20260806 20:41:02.658780 6335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:02.660761 6344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:41:02.664592 6344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:41:02.672112 6344 catalog_manager.cc:1347] Generated new cluster ID: c1bcb9c3478e4ce89d3be48ce2ad4b8e
I20260806 20:41:02.672217 6344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:41:02.698823 6344 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:41:02.700556 6344 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:41:02.709671 6344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 065a967a6c4243c19bb33eff0699acd2: Generated new TSK 0
I20260806 20:41:02.710484 6344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:41:02.757922 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6261
I20260806 20:41:02.759130 6324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:02.857246 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6261
2026-08-06T20:41:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2015 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:41:02 dist-test-slave-tm1g krb5kdc[6358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:41:02 dist-test-slave-tm1g krb5kdc[6358](info): set up 2 sockets
Aug 06 20:41:02 dist-test-slave-tm1g krb5kdc[6358](info): commencing operation
krb5kdc: starting...
W20260806 20:41:03.974735 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:41:04 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:41:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:41:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:41:04.201371 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35589
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:04.529975 6371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:04.530231 6371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:04.530412 6371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:04.537474 6371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:41:04.537566 6371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:04.537609 6371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:04.537685 6371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:41:04.537771 6371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:41:04.547324 6371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:04.548540 6371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:04.549830 6371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:04.561399 6377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:04.561340 6379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:04.561437 6376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:04.562390 6371 server_base.cc:1034] running on GCE node
I20260806 20:41:04.563235 6371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:04.564581 6371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:04.565812 6371 hybrid_clock.cc:648] HybridClock initialized: now 1786048864565761 us; error 52 us; skew 500 ppm
I20260806 20:41:04.566156 6371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:04 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048864, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:04.573396 6371 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:41:04.574819 6371 webserver.cc:460] Webserver started at http://127.23.200.62:35015/ using document root <none> and password file <none>
I20260806 20:41:04.575439 6371 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:04.575541 6371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:04.575809 6371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:04.578733 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "5a3ff2e24f054334a20b77cc78026be4"
format_stamp: "Formatted at 2026-08-06 20:41:04 on dist-test-slave-tm1g"
server_key: "93b10fc1edb313c89cfd5d018436a09c"
server_key_iv: "f8753f65f298db887a576be0a7f3c340"
server_key_version: "encryptionkey@0"
I20260806 20:41:04.579432 6371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "5a3ff2e24f054334a20b77cc78026be4"
format_stamp: "Formatted at 2026-08-06 20:41:04 on dist-test-slave-tm1g"
server_key: "93b10fc1edb313c89cfd5d018436a09c"
server_key_iv: "f8753f65f298db887a576be0a7f3c340"
server_key_version: "encryptionkey@0"
I20260806 20:41:04.584527 6371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:41:04.587798 6386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:04.589087 6371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:41:04.589267 6371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "5a3ff2e24f054334a20b77cc78026be4"
format_stamp: "Formatted at 2026-08-06 20:41:04 on dist-test-slave-tm1g"
server_key: "93b10fc1edb313c89cfd5d018436a09c"
server_key_iv: "f8753f65f298db887a576be0a7f3c340"
server_key_version: "encryptionkey@0"
I20260806 20:41:04.589457 6371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:04.605983 6371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:04.609839 6371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:04.610146 6371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:04.628764 6371 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:35589
I20260806 20:41:04.628876 6439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:35589 every 8 connection(s)
I20260806 20:41:04.630105 6371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:41:04.633706 6440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:04.634166 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6371
I20260806 20:41:04.634438 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:41:04.635298 24352 external_mini_cluster.cc:1428] Setting key b99b25ebc79939e2b6d7772bae1c8ab6
I20260806 20:41:04.648099 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap starting.
Aug 06 20:41:04 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:04.651712 6440 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:04.652874 6440 log.cc:824] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:04.656055 6440 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: No bootstrap required, opened a new log
I20260806 20:41:04.659827 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:04.637332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.23.200.62:35589)
0806 20:41:04.638114 (+ 782us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:04.638125 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:04.638170 (+ 45us) server_negotiation.cc:407] Connection header received
0806 20:41:04.640105 (+ 1935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:04.640149 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:04.640542 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:04.640868 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:04.642644 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:04.643940 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:04.645900 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:04.646651 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:04.646793 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:04.651725 (+ 4932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:04.651732 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:04.651745 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:04.651813 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:04.655374 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:04.656058 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:04.656068 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:04.656078 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:04.656273 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:04.656654 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:04.656658 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:04.656662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:04.657439 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:04.657761 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:04.658036 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 20:41:04.658523 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":360,"threads_started":1}
I20260806 20:41:04.665536 6440 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:04.665843 6440 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:04.665915 6440 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3ff2e24f054334a20b77cc78026be4, State: Initialized, Role: FOLLOWER
I20260806 20:41:04.666430 6440 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:04.666630 6440 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:41:04.666725 6440 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:41:04.666859 6440 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:04.668843 6440 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:04.669238 6440 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3ff2e24f054334a20b77cc78026be4; no voters:
I20260806 20:41:04.669744 6440 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:41:04.669919 6445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:04.670609 6445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 LEADER]: Becoming Leader. State: Replica: 5a3ff2e24f054334a20b77cc78026be4, State: Running, Role: LEADER
I20260806 20:41:04.671136 6445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:04.671936 6440 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:41:04.674094 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a3ff2e24f054334a20b77cc78026be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } } }
I20260806 20:41:04.674439 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:04.674504 6447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3ff2e24f054334a20b77cc78026be4. Latest consensus state: current_term: 1 leader_uuid: "5a3ff2e24f054334a20b77cc78026be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } } }
I20260806 20:41:04.674810 6447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:04.676496 6451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:41:04.681108 6451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:41:04.690074 6451 catalog_manager.cc:1347] Generated new cluster ID: e44719e0728d4b5d97ce2b0fb3ed1d28
I20260806 20:41:04.690188 6451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:41:04.718673 6451 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:41:04.719915 6451 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:41:04.731515 6451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Generated new TSK 0
I20260806 20:41:04.732335 6451 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:41:04.799304 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35589
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:05.130386 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:05.130645 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:05.130822 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:05.137789 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:05.137902 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:05.138047 6466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:41:05.147464 6466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:05.148665 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:05.149951 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:05.162902 6475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:05.162871 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:05.163467 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:05.164445 6466 server_base.cc:1034] running on GCE node
I20260806 20:41:05.164974 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:05.165908 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:05.167106 6466 hybrid_clock.cc:648] HybridClock initialized: now 1786048865167063 us; error 31 us; skew 500 ppm
I20260806 20:41:05.167371 6466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:05 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048865, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:05.170392 6466 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:41:05.171836 6466 webserver.cc:460] Webserver started at http://127.23.200.1:33877/ using document root <none> and password file <none>
I20260806 20:41:05.172554 6466 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:05.172698 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:05.172999 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:05.175961 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "1c74e0674f4a4032bdeb10ac71661107"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "2555827197be919af84585ed920644e1"
server_key_iv: "7b7f21351e6671be0df767b571ec110a"
server_key_version: "encryptionkey@0"
I20260806 20:41:05.176774 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "1c74e0674f4a4032bdeb10ac71661107"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "2555827197be919af84585ed920644e1"
server_key_iv: "7b7f21351e6671be0df767b571ec110a"
server_key_version: "encryptionkey@0"
I20260806 20:41:05.182066 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:41:05.185442 6482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:05.186748 6466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:05.186919 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "1c74e0674f4a4032bdeb10ac71661107"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "2555827197be919af84585ed920644e1"
server_key_iv: "7b7f21351e6671be0df767b571ec110a"
server_key_version: "encryptionkey@0"
I20260806 20:41:05.187127 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:05.197579 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:05.201267 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:05.201514 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:05.202387 6466 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:05.203645 6466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:05.203716 6466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:05.203797 6466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:05.203845 6466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:05.236722 6466 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:37675
I20260806 20:41:05.236770 6597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:37675 every 8 connection(s)
I20260806 20:41:05.237983 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:41:05.241263 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6466
I20260806 20:41:05.241487 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:41:05.242170 24352 external_mini_cluster.cc:1428] Setting key 0f7fa85bbd94bbb0d26fafc7b82c6ecb
Aug 06 20:41:05 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048865, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:05.257896 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:05.240570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:36357 (local address 127.23.200.62:35589)
0806 20:41:05.241097 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:05.241104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:05.242168 (+ 1064us) server_negotiation.cc:407] Connection header received
0806 20:41:05.244200 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:05.244205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:05.244295 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:05.244419 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:05.245874 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.246700 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.248293 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.248691 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.248830 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:05.252245 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:05.252250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:05.252253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:05.252283 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:05.254900 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:05.255607 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:05.255618 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:05.255621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:05.255738 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:05.256165 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:05.256173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:05.256176 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:05.256443 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:05.256597 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:05.257284 (+ 687us) server_negotiation.cc:299] Negotiation successful
0806 20:41:05.257545 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":241,"threads_started":1}
I20260806 20:41:05.258843 6601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:05.241148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:36357)
0806 20:41:05.241982 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:05.242026 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:05.243937 (+ 1911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:05.244645 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:05.244652 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:05.245152 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:05.245707 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:05.245724 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.246863 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.246886 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:05.248132 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:05.248150 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.248928 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.248936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:05.249166 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:05.249846 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:05.249866 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:05.252041 (+ 2175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:05.255038 (+ 2997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:05.255046 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:05.255060 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:05.255476 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:05.255852 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:05.255856 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:05.255860 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:05.256048 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:05.256572 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:05.256580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:05.256893 (+ 313us) client_negotiation.cc:769] Sending connection context
0806 20:41:05.257182 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 20:41:05.257697 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":370,"threads_started":1}
I20260806 20:41:05.260393 6598 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35589
I20260806 20:41:05.260725 6598 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:05.261420 6598 heartbeater.cc:511] Master 127.23.200.62:35589 requested a full tablet report, sending...
I20260806 20:41:05.263801 6404 ts_manager.cc:194] Registered new tserver with Master: 1c74e0674f4a4032bdeb10ac71661107 (127.23.200.1:37675)
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
I20260806 20:41:05.266050 6404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:36357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:41:05.295750 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35589
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:05.621994 6604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:05.622239 6604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:05.622349 6604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:05.629315 6604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:05.629407 6604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:05.629577 6604 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:41:05.638974 6604 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:05.640125 6604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:05.641508 6604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:05.654279 6609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:05.654230 6612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:05.654346 6610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:05.655020 6604 server_base.cc:1034] running on GCE node
I20260806 20:41:05.655566 6604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:05.656415 6604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:05.657635 6604 hybrid_clock.cc:648] HybridClock initialized: now 1786048865657587 us; error 58 us; skew 500 ppm
I20260806 20:41:05.657932 6604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:05 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048865, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:05.660952 6604 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:41:05.662221 6604 webserver.cc:460] Webserver started at http://127.23.200.2:41443/ using document root <none> and password file <none>
I20260806 20:41:05.662869 6604 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:05.662977 6604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:05.663223 6604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:05.665974 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "61051e54a4e0490c9f7385cfd3667f87"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "9a4c67319f2f4cffe2508780d5a0aae6"
server_key_iv: "3b051bdc0e69bc754161d64abe41d0a8"
server_key_version: "encryptionkey@0"
I20260806 20:41:05.666733 6604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "61051e54a4e0490c9f7385cfd3667f87"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "9a4c67319f2f4cffe2508780d5a0aae6"
server_key_iv: "3b051bdc0e69bc754161d64abe41d0a8"
server_key_version: "encryptionkey@0"
I20260806 20:41:05.671490 6604 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260806 20:41:05.674961 6619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:05.676163 6604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:41:05.676324 6604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "61051e54a4e0490c9f7385cfd3667f87"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "9a4c67319f2f4cffe2508780d5a0aae6"
server_key_iv: "3b051bdc0e69bc754161d64abe41d0a8"
server_key_version: "encryptionkey@0"
I20260806 20:41:05.676513 6604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:05.699718 6604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:05.703481 6604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:05.703763 6604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:05.704672 6604 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:05.705943 6604 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:05.706024 6604 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:05.706190 6604 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:05.706241 6604 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:05.738936 6604 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:32837
I20260806 20:41:05.738999 6734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:32837 every 8 connection(s)
I20260806 20:41:05.740537 6604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:41:05.747045 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6604
I20260806 20:41:05.747259 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:41:05.747848 24352 external_mini_cluster.cc:1428] Setting key b0664d1bb50566d5c87aadaaff8a80cc
Aug 06 20:41:05 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048865, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:05.763546 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:05.743850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:51861 (local address 127.23.200.62:35589)
0806 20:41:05.744556 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:05.744563 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:05.746355 (+ 1792us) server_negotiation.cc:407] Connection header received
0806 20:41:05.748888 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:05.748895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:05.749004 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:05.749115 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:05.751059 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.751768 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.753279 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.753724 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.753856 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:05.757600 (+ 3744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:05.757605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:05.757608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:05.757641 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:05.760559 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:05.761267 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:05.761272 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:05.761274 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:05.761394 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:05.761818 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:05.761823 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:05.761826 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:05.762092 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:05.762246 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:05.762984 (+ 738us) server_negotiation.cc:299] Negotiation successful
0806 20:41:05.763210 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546}
I20260806 20:41:05.764523 6737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:05.744418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:51861)
0806 20:41:05.746170 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:05.746215 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:05.748594 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:05.749528 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:05.749542 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:05.750257 (+ 715us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:05.750885 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:05.750902 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.751919 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.751925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:05.753131 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:05.753142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:05.753964 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:05.753971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:05.754198 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:05.754926 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:05.754956 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:05.757372 (+ 2416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:05.760694 (+ 3322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:05.760702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:05.760718 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:05.761134 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:05.761506 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:05.761511 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:05.761515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:05.761700 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:05.762225 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:05.762233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:05.762603 (+ 370us) client_negotiation.cc:769] Sending connection context
0806 20:41:05.762865 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 20:41:05.763380 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1477,"thread_start_us":1251,"threads_started":1}
I20260806 20:41:05.766037 6735 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35589
I20260806 20:41:05.766398 6735 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:05.767119 6735 heartbeater.cc:511] Master 127.23.200.62:35589 requested a full tablet report, sending...
I20260806 20:41:05.768659 6404 ts_manager.cc:194] Registered new tserver with Master: 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837)
I20260806 20:41:05.770016 6404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:51861
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:41:05.801621 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35589
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:06.125649 6740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:06.125902 6740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:06.126063 6740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:06.133175 6740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:06.133266 6740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:06.133433 6740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:41:06.142882 6740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:06.144093 6740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:06.145418 6740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:06.158207 6748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:06.158326 6746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:06.158316 6745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:06.159334 6740 server_base.cc:1034] running on GCE node
I20260806 20:41:06.159945 6740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:06.160848 6740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:06.162075 6740 hybrid_clock.cc:648] HybridClock initialized: now 1786048866162020 us; error 47 us; skew 500 ppm
I20260806 20:41:06.162429 6740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:06 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:06.165349 6740 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:41:06.166719 6740 webserver.cc:460] Webserver started at http://127.23.200.3:36197/ using document root <none> and password file <none>
I20260806 20:41:06.167380 6740 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:06.167526 6740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:06.167821 6740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:06.170756 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "0e97422439df4f5b89ef871fc5baa004"
format_stamp: "Formatted at 2026-08-06 20:41:06 on dist-test-slave-tm1g"
server_key: "a5d06d7f655268efb50e12732ddc02a5"
server_key_iv: "031aaf66befae33b8534a52788180622"
server_key_version: "encryptionkey@0"
I20260806 20:41:06.171526 6740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "0e97422439df4f5b89ef871fc5baa004"
format_stamp: "Formatted at 2026-08-06 20:41:06 on dist-test-slave-tm1g"
server_key: "a5d06d7f655268efb50e12732ddc02a5"
server_key_iv: "031aaf66befae33b8534a52788180622"
server_key_version: "encryptionkey@0"
I20260806 20:41:06.176626 6740 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:41:06.180104 6755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:06.181417 6740 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:41:06.181605 6740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "0e97422439df4f5b89ef871fc5baa004"
format_stamp: "Formatted at 2026-08-06 20:41:06 on dist-test-slave-tm1g"
server_key: "a5d06d7f655268efb50e12732ddc02a5"
server_key_iv: "031aaf66befae33b8534a52788180622"
server_key_version: "encryptionkey@0"
I20260806 20:41:06.181808 6740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:06.196228 6740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:06.199949 6740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:06.200234 6740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:06.201144 6740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:06.202466 6740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:06.202572 6740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:06.202724 6740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:06.202797 6740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:06.235785 6740 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:36023
I20260806 20:41:06.235827 6871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:36023 every 8 connection(s)
I20260806 20:41:06.237080 6740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:41:06.244391 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6740
I20260806 20:41:06.244597 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:41:06.245030 24352 external_mini_cluster.cc:1428] Setting key 8ffa47554f7842c59f24385907f6288f
Aug 06 20:41:06 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048866, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:06.257261 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.239896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:60871 (local address 127.23.200.62:35589)
0806 20:41:06.240120 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.240126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.241855 (+ 1729us) server_negotiation.cc:407] Connection header received
0806 20:41:06.243473 (+ 1618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.243478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.243550 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.243630 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:06.245133 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.246388 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.248549 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.248979 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.249079 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.252367 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:06.252372 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:06.252375 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:06.252406 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:06.254840 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.255455 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.255458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.255461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.255582 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.255936 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.255939 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.255941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.256181 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:06.256288 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.256864 (+ 576us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.257028 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:41:06.258157 6875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.240263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:60871)
0806 20:41:06.241092 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:06.241141 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:06.243222 (+ 2081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:06.243821 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:06.243829 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:06.244303 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:06.244958 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.244975 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.246584 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.246588 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.248404 (+ 1816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.248420 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.249163 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.249168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.249361 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.250021 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:06.250043 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:06.252182 (+ 2139us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:06.254961 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:06.254967 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:06.254978 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:06.255359 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:06.255671 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:06.255674 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:06.255676 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:06.255843 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:06.256282 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:06.256288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:06.256561 (+ 273us) client_negotiation.cc:769] Sending connection context
0806 20:41:06.256774 (+ 213us) client_negotiation.cc:240] Negotiation successful
0806 20:41:06.257127 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":365,"threads_started":1}
I20260806 20:41:06.259667 6872 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35589
I20260806 20:41:06.259997 6872 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:06.260782 6872 heartbeater.cc:511] Master 127.23.200.62:35589 requested a full tablet report, sending...
I20260806 20:41:06.262137 6404 ts_manager.cc:194] Registered new tserver with Master: 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
I20260806 20:41:06.262986 6404 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:60871
I20260806 20:41:06.268920 6598 heartbeater.cc:503] Master 127.23.200.62:35589 was elected leader, sending a full tablet report...
I20260806 20:41:06.272624 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:41:06.289505 6600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.277172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.23.200.62:35589)
0806 20:41:06.277347 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.277355 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.277827 (+ 472us) server_negotiation.cc:407] Connection header received
0806 20:41:06.278005 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.278008 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.278066 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.278143 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:06.279519 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.280311 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.281689 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.282102 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.282195 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.284604 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:06.284609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:06.284611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:06.284641 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:06.287254 (+ 2613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.287914 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.287918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.287921 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.288046 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.288366 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.288369 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.288371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.288679 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:06.288859 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.289118 (+ 259us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.289271 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 20:41:06.293432 6404 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:41:06.300354 6404 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:41:06.336889 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.312256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35962 (local address 127.23.200.1:37675)
0806 20:41:06.327591 (+ 15335us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.327605 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.327648 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:41:06.327777 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.327789 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.328125 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.328377 (+ 252us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:06.330265 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.331826 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.334087 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.335646 (+ 1559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.335788 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.336026 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.336358 (+ 332us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.336537 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15067,"thread_start_us":16834,"threads_started":1}
I20260806 20:41:06.336920 6885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.311951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35962)
0806 20:41:06.326260 (+ 14309us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:06.326337 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:06.326588 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:06.329156 (+ 2568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:06.329164 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:06.329199 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:06.330050 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.330070 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.331969 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.331973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.333938 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.333950 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.335876 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.335883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.336106 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.336160 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 20:41:06.336292 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 20:41:06.336478 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13938,"mutex_wait_us":35,"thread_start_us":14858,"threads_started":1}
I20260806 20:41:06.338126 6889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.312115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32837 (local address 127.0.0.1:49202)
0806 20:41:06.329909 (+ 17794us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:06.329944 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:06.330105 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:06.330747 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:06.330752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:06.330775 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:06.331175 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.331188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.333009 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.333014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.335277 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.335292 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.337313 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.337319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.337491 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.337514 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:41:06.337620 (+ 106us) client_negotiation.cc:240] Negotiation successful
0806 20:41:06.337745 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17634,"mutex_wait_us":24,"thread_start_us":17626,"threads_started":1}
I20260806 20:41:06.338155 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.312460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49202 (local address 127.23.200.2:32837)
0806 20:41:06.329086 (+ 16626us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.329094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.330059 (+ 965us) server_negotiation.cc:407] Connection header received
0806 20:41:06.330398 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.330406 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.330569 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.330757 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:06.331368 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.332861 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.335428 (+ 2567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.336795 (+ 1367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.336964 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.337164 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.337664 (+ 500us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.337830 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16467,"thread_start_us":16340,"threads_started":1}
I20260806 20:41:06.340854 6532 tablet_service.cc:1467] Processing CreateTablet for tablet e6a8f476af75462f95dcd5eeb7c2ea38 (DEFAULT_TABLE table=test-table [id=102f0833c1a6467ea894ec0150b60ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:06.341486 6668 tablet_service.cc:1467] Processing CreateTablet for tablet e6a8f476af75462f95dcd5eeb7c2ea38 (DEFAULT_TABLE table=test-table [id=102f0833c1a6467ea894ec0150b60ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:06.342175 6532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a8f476af75462f95dcd5eeb7c2ea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:06.342861 6668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a8f476af75462f95dcd5eeb7c2ea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:06.342901 6886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.314030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:36023 (local address 127.0.0.1:48896)
0806 20:41:06.330069 (+ 16039us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:06.330095 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:06.330225 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:06.332506 (+ 2281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:06.332511 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:06.332536 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:06.333052 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.333065 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.335102 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.335109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.337115 (+ 2006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.337129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.342125 (+ 4996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.342133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.342347 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.342380 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 20:41:06.342464 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 20:41:06.342601 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15870,"thread_start_us":15498,"threads_started":1}
I20260806 20:41:06.343283 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.314766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48896 (local address 127.23.200.3:36023)
0806 20:41:06.329137 (+ 14371us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.329148 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.332003 (+ 2855us) server_negotiation.cc:407] Connection header received
0806 20:41:06.332106 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.332113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.332316 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.332494 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:06.333269 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.334901 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.337309 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.341911 (+ 4602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.342054 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.342491 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.342568 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.342724 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14181,"thread_start_us":14015,"threads_started":1}
I20260806 20:41:06.346747 6806 tablet_service.cc:1467] Processing CreateTablet for tablet e6a8f476af75462f95dcd5eeb7c2ea38 (DEFAULT_TABLE table=test-table [id=102f0833c1a6467ea894ec0150b60ce8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:06.348096 6806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6a8f476af75462f95dcd5eeb7c2ea38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:06.355170 6891 tablet_bootstrap.cc:492] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Bootstrap starting.
I20260806 20:41:06.358785 6891 tablet_bootstrap.cc:654] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:06.359043 6892 tablet_bootstrap.cc:492] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Bootstrap starting.
I20260806 20:41:06.360610 6891 log.cc:824] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:06.363003 6892 tablet_bootstrap.cc:654] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:06.363904 6891 tablet_bootstrap.cc:492] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: No bootstrap required, opened a new log
I20260806 20:41:06.364177 6891 ts_tablet_manager.cc:1397] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 20:41:06.364250 6892 log.cc:824] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:06.366649 6894 tablet_bootstrap.cc:492] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Bootstrap starting.
I20260806 20:41:06.367203 6892 tablet_bootstrap.cc:492] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: No bootstrap required, opened a new log
I20260806 20:41:06.367461 6892 ts_tablet_manager.cc:1397] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 20:41:06.370072 6894 tablet_bootstrap.cc:654] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:06.371254 6894 log.cc:824] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:06.372043 6891 raft_consensus.cc:348] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.372584 6891 raft_consensus.cc:374] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:06.372725 6891 raft_consensus.cc:729] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61051e54a4e0490c9f7385cfd3667f87, State: Initialized, Role: FOLLOWER
I20260806 20:41:06.373405 6891 consensus_queue.cc:260] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.374596 6735 heartbeater.cc:503] Master 127.23.200.62:35589 was elected leader, sending a full tablet report...
I20260806 20:41:06.375278 6891 ts_tablet_manager.cc:1428] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260806 20:41:06.375560 6894 tablet_bootstrap.cc:492] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: No bootstrap required, opened a new log
I20260806 20:41:06.375820 6894 ts_tablet_manager.cc:1397] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 20:41:06.378350 6892 raft_consensus.cc:348] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.378785 6892 raft_consensus.cc:374] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:06.378895 6892 raft_consensus.cc:729] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c74e0674f4a4032bdeb10ac71661107, State: Initialized, Role: FOLLOWER
I20260806 20:41:06.379496 6892 consensus_queue.cc:260] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.381314 6892 ts_tablet_manager.cc:1428] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 20:41:06.384438 6894 raft_consensus.cc:348] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.384837 6894 raft_consensus.cc:374] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:06.384989 6894 raft_consensus.cc:729] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e97422439df4f5b89ef871fc5baa004, State: Initialized, Role: FOLLOWER
I20260806 20:41:06.385628 6894 consensus_queue.cc:260] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.386690 6872 heartbeater.cc:503] Master 127.23.200.62:35589 was elected leader, sending a full tablet report...
I20260806 20:41:06.387110 6894 ts_tablet_manager.cc:1428] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 20:41:06.389746 6898 raft_consensus.cc:482] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:06.390031 6898 raft_consensus.cc:504] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.391747 6898 leader_election.cc:288] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
I20260806 20:41:06.398443 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.392423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:48025 (local address 127.23.200.3:36023)
0806 20:41:06.392626 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.392633 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.392920 (+ 287us) server_negotiation.cc:407] Connection header received
0806 20:41:06.393143 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.393150 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.393235 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.393321 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:06.393993 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.394908 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.396696 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.397448 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.397632 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.397706 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.398051 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.398175 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 20:41:06.398556 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.392113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49317 (local address 127.23.200.2:32837)
0806 20:41:06.392627 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.392632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.393254 (+ 622us) server_negotiation.cc:407] Connection header received
0806 20:41:06.393415 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.393422 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.393513 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.393609 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:06.394341 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.395107 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.396817 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.397588 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.397765 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.397842 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.398143 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.398269 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371}
I20260806 20:41:06.398638 6901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.392109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32837 (local address 127.23.200.1:49317)
0806 20:41:06.393143 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:06.393165 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:06.393289 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:06.393656 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:06.393661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:06.393684 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:06.394121 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.394133 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.395254 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.395262 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.396663 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.396674 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.397805 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.397812 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.397994 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.398016 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:41:06.398118 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 20:41:06.398238 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":904,"mutex_wait_us":22,"thread_start_us":702,"threads_started":1}
I20260806 20:41:06.399153 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.391980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:36023 (local address 127.23.200.1:48025)
0806 20:41:06.392781 (+ 801us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:06.392810 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:06.392978 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:06.393334 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:06.393341 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:06.393372 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:06.393867 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.393878 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.395064 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.395070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.396530 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:06.396542 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.397652 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.397658 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:06.397886 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.397914 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 20:41:06.398015 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 20:41:06.398140 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":24,"thread_start_us":337,"threads_started":1}
I20260806 20:41:06.399638 6688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a8f476af75462f95dcd5eeb7c2ea38" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61051e54a4e0490c9f7385cfd3667f87" is_pre_election: true
I20260806 20:41:06.400070 6688 raft_consensus.cc:2454] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 0.
I20260806 20:41:06.400079 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a8f476af75462f95dcd5eeb7c2ea38" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e97422439df4f5b89ef871fc5baa004" is_pre_election: true
I20260806 20:41:06.400566 6826 raft_consensus.cc:2454] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 0.
I20260806 20:41:06.400718 6486 leader_election.cc:302] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107, 61051e54a4e0490c9f7385cfd3667f87; no voters:
I20260806 20:41:06.401134 6898 raft_consensus.cc:2781] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:41:06.401294 6898 raft_consensus.cc:482] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:41:06.401413 6898 raft_consensus.cc:3037] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:06.403596 6898 raft_consensus.cc:504] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.404177 6898 leader_election.cc:288] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 election: Requested vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
I20260806 20:41:06.404527 6688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a8f476af75462f95dcd5eeb7c2ea38" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61051e54a4e0490c9f7385cfd3667f87"
I20260806 20:41:06.404631 6826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6a8f476af75462f95dcd5eeb7c2ea38" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e97422439df4f5b89ef871fc5baa004"
I20260806 20:41:06.404716 6688 raft_consensus.cc:3037] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:06.404844 6826 raft_consensus.cc:3037] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:06.407330 6688 raft_consensus.cc:2454] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 1.
I20260806 20:41:06.407371 6826 raft_consensus.cc:2454] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 1.
I20260806 20:41:06.407768 6485 leader_election.cc:302] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e97422439df4f5b89ef871fc5baa004, 1c74e0674f4a4032bdeb10ac71661107; no voters:
I20260806 20:41:06.408113 6898 raft_consensus.cc:2781] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:06.408499 6898 raft_consensus.cc:686] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 LEADER]: Becoming Leader. State: Replica: 1c74e0674f4a4032bdeb10ac71661107, State: Running, Role: LEADER
I20260806 20:41:06.409027 6898 consensus_queue.cc:237] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:06.413371 6404 catalog_manager.cc:5495] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c74e0674f4a4032bdeb10ac71661107 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "1c74e0674f4a4032bdeb10ac71661107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } health_report { overall_health: UNKNOWN } } }
I20260806 20:41:06.436602 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.431567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.23.200.1:37675)
0806 20:41:06.431755 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.431760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.431868 (+ 108us) server_negotiation.cc:407] Connection header received
0806 20:41:06.432033 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.432037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.432110 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.432230 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:41:06.432576 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.433290 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.434696 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.435093 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.435183 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.435624 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:41:06.436163 (+ 539us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:41:06.436275 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.436352 (+ 77us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.436412 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:41:06.444355 6826 raft_consensus.cc:1261] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Refusing update from remote peer 1c74e0674f4a4032bdeb10ac71661107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:06.444382 6688 raft_consensus.cc:1261] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Refusing update from remote peer 1c74e0674f4a4032bdeb10ac71661107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:06.445307 6902 consensus_queue.cc:1035] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:06.445705 6898 consensus_queue.cc:1035] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:06.459738 6906 mvcc.cc:205] Tried to move back new op lower bound from 7315656156946243584 to 7315656156815011840. Current Snapshot: MvccSnapshot[applied={T|T < 7315656156946243584}]
I20260806 20:41:06.470463 6403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:06.470798 6403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:06.474684 6403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Sending DeleteTablet for 3 replicas of tablet e6a8f476af75462f95dcd5eeb7c2ea38
I20260806 20:41:06.475905 6532 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a8f476af75462f95dcd5eeb7c2ea38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:06 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35962
I20260806 20:41:06.475960 6806 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a8f476af75462f95dcd5eeb7c2ea38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:06 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:48896
I20260806 20:41:06.476022 6668 tablet_service.cc:1514] Processing DeleteTablet for tablet e6a8f476af75462f95dcd5eeb7c2ea38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:06 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:49202
I20260806 20:41:06.477531 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6466
I20260806 20:41:06.485267 6917 tablet_replica.cc:330] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: stopping tablet replica
I20260806 20:41:06.485989 6917 raft_consensus.cc:2229] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:06.486560 6917 raft_consensus.cc:2258] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:06.488782 6917 ts_tablet_manager.cc:1905] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:06.493048 6915 tablet_replica.cc:330] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: stopping tablet replica
I20260806 20:41:06.493458 6915 raft_consensus.cc:2229] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:06.494014 6915 raft_consensus.cc:2258] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:06.494798 6916 tablet_replica.cc:330] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: stopping tablet replica
I20260806 20:41:06.495187 6916 raft_consensus.cc:2229] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:41:06.495816 6916 raft_consensus.cc:2258] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:06.497923 6915 ts_tablet_manager.cc:1905] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:06.499269 6916 ts_tablet_manager.cc:1905] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:06.504873 6915 ts_tablet_manager.cc:1918] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:06.505038 6915 log.cc:1196] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/e6a8f476af75462f95dcd5eeb7c2ea38
I20260806 20:41:06.505256 6916 ts_tablet_manager.cc:1918] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:06.505431 6916 log.cc:1196] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/e6a8f476af75462f95dcd5eeb7c2ea38
I20260806 20:41:06.505633 6915 ts_tablet_manager.cc:1939] T e6a8f476af75462f95dcd5eeb7c2ea38 P 0e97422439df4f5b89ef871fc5baa004: Deleting consensus metadata
Aug 06 20:41:06 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:41:06.506006 6916 ts_tablet_manager.cc:1939] T e6a8f476af75462f95dcd5eeb7c2ea38 P 1c74e0674f4a4032bdeb10ac71661107: Deleting consensus metadata
I20260806 20:41:06.507496 6390 catalog_manager.cc:4841] TS 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): tablet e6a8f476af75462f95dcd5eeb7c2ea38 (table test-table [id=102f0833c1a6467ea894ec0150b60ce8]) successfully deleted
I20260806 20:41:06.507651 6388 catalog_manager.cc:4841] TS 1c74e0674f4a4032bdeb10ac71661107 (127.23.200.1:37675): tablet e6a8f476af75462f95dcd5eeb7c2ea38 (table test-table [id=102f0833c1a6467ea894ec0150b60ce8]) successfully deleted
I20260806 20:41:06.510530 6917 ts_tablet_manager.cc:1918] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:06.510697 6917 log.cc:1196] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/e6a8f476af75462f95dcd5eeb7c2ea38
I20260806 20:41:06.511273 6917 ts_tablet_manager.cc:1939] T e6a8f476af75462f95dcd5eeb7c2ea38 P 61051e54a4e0490c9f7385cfd3667f87: Deleting consensus metadata
I20260806 20:41:06.513036 6391 catalog_manager.cc:4841] TS 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): tablet e6a8f476af75462f95dcd5eeb7c2ea38 (table test-table [id=102f0833c1a6467ea894ec0150b60ce8]) successfully deleted
I20260806 20:41:06.515913 6887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.478683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35976 (local address 127.23.200.1:37675)
0806 20:41:06.478908 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.478918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.485868 (+ 6950us) server_negotiation.cc:407] Connection header received
0806 20:41:06.486074 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.486082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.486200 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.486505 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:06.494209 (+ 7704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.495302 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.497148 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.497621 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.497772 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.508736 (+ 10964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:06.508749 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:06.508765 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:06.508811 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:06.512966 (+ 4155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.513720 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.513727 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.513730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.513897 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.514343 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.514349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.514353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.514707 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:06.515030 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.515319 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.515560 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 20:41:06.516580 6593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:06.616384 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6466
I20260806 20:41:06.636721 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6604
Aug 06 20:41:06 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:41:06.653039 6888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.637532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49214 (local address 127.23.200.2:32837)
0806 20:41:06.637718 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.637724 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.637745 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:41:06.637913 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.637919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.637997 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.638114 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:06.639672 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.640842 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.642155 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.642782 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.642928 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.647345 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:06.647361 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:06.647378 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:06.647418 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:06.650079 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.650853 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.650861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.650863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.651003 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.651415 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.651420 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.651422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.651796 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:06.652128 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.652467 (+ 339us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.652695 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:41:06.653729 6730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:06.743090 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6604
I20260806 20:41:06.765400 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6740
Aug 06 20:41:06 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048864, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:41:06.781268 6890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:06.766202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48906 (local address 127.23.200.3:36023)
0806 20:41:06.766415 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:06.766422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:06.766524 (+ 102us) server_negotiation.cc:407] Connection header received
0806 20:41:06.766754 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:06.766758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:06.766840 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:06.766949 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:06.768258 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.769002 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.770353 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:06.770766 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:06.770913 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:06.775518 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:06.775535 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:06.775553 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:06.775586 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:06.778531 (+ 2945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.779180 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.779191 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.779199 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.779340 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:06.779712 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:06.779717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:06.779720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:06.780061 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:06.780360 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:06.780707 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 20:41:06.780950 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:41:06.781936 6867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:06.872011 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6740
I20260806 20:41:06.890527 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6371
I20260806 20:41:06.891893 6435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:07.002519 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6371
I20260806 20:41:07.024953 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35589
--webserver_interface=127.23.200.62
--webserver_port=35015
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:07.351856 6923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:07.352133 6923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:07.352288 6923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:07.359515 6923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:41:07.359620 6923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:07.359706 6923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:07.359812 6923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:41:07.359894 6923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:41:07.369652 6923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=35015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:07.371019 6923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:07.372311 6923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:07.395399 6930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:07.395498 6929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:07.395471 6932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:07.396466 6923 server_base.cc:1034] running on GCE node
I20260806 20:41:07.397416 6923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:07.399060 6923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:07.400309 6923 hybrid_clock.cc:648] HybridClock initialized: now 1786048867400257 us; error 56 us; skew 500 ppm
I20260806 20:41:07.400718 6923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:07 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048867, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:07.404210 6923 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:41:07.405879 6923 webserver.cc:460] Webserver started at http://127.23.200.62:35015/ using document root <none> and password file <none>
I20260806 20:41:07.406708 6923 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:07.406874 6923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:07.412715 6923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:41:07.415544 6939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:07.416857 6923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:07.417047 6923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "5a3ff2e24f054334a20b77cc78026be4"
format_stamp: "Formatted at 2026-08-06 20:41:04 on dist-test-slave-tm1g"
server_key: "93b10fc1edb313c89cfd5d018436a09c"
server_key_iv: "f8753f65f298db887a576be0a7f3c340"
server_key_version: "encryptionkey@0"
I20260806 20:41:07.418171 6923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:07.437489 6923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:07.441494 6923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:07.441782 6923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:07.460878 6923 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:35589
I20260806 20:41:07.460897 6991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:35589 every 8 connection(s)
I20260806 20:41:07.462380 6923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:41:07.468837 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6923
I20260806 20:41:07.469379 6992 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:41:07.469709 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:37675
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:35589
--webserver_port=33877
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:41:07.472736 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap starting.
I20260806 20:41:07.493276 6992 log.cc:824] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:07.504959 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:41:07.505505 6992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap complete.
I20260806 20:41:07.514607 6992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:07.515445 6992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3ff2e24f054334a20b77cc78026be4, State: Initialized, Role: FOLLOWER
I20260806 20:41:07.516044 6992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:07.516292 6992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:41:07.516425 6992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:41:07.516605 6992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:41:07.520375 6992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:07.520871 6992 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3ff2e24f054334a20b77cc78026be4; no voters:
I20260806 20:41:07.521515 6992 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 20:41:07.521660 6996 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:41:07.524221 6992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:41:07.528611 6996 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 LEADER]: Becoming Leader. State: Replica: 5a3ff2e24f054334a20b77cc78026be4, State: Running, Role: LEADER
I20260806 20:41:07.529202 6996 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:07.534171 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5a3ff2e24f054334a20b77cc78026be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } } }
I20260806 20:41:07.534623 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:07.535208 6998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3ff2e24f054334a20b77cc78026be4. Latest consensus state: current_term: 2 leader_uuid: "5a3ff2e24f054334a20b77cc78026be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } } }
I20260806 20:41:07.535486 6998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:07.536569 7012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:41:07.544601 7012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:41:07.546453 7012 catalog_manager.cc:1259] Loaded cluster ID: e44719e0728d4b5d97ce2b0fb3ed1d28
I20260806 20:41:07.546635 7012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:41:07.550688 7012 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 20:41:07.564651 7012 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 20:41:07.571731 7012 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:41:07.572996 7012 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Loaded TSK: 0
I20260806 20:41:07.574070 7012 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 20:41:07.827293 6994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:07.827600 6994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:07.827766 6994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:07.835009 6994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:07.835147 6994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:07.835317 6994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:41:07.846679 6994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:37675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=33877
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:07.847990 6994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:07.849309 6994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:07.865895 7024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:07.866008 7023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:07.866801 7026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:07.866981 6994 server_base.cc:1034] running on GCE node
I20260806 20:41:07.867594 6994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:07.868587 6994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:07.869812 6994 hybrid_clock.cc:648] HybridClock initialized: now 1786048867869761 us; error 43 us; skew 500 ppm
I20260806 20:41:07.870121 6994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:07 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048867, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:07.873526 6994 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:41:07.875290 6994 webserver.cc:460] Webserver started at http://127.23.200.1:33877/ using document root <none> and password file <none>
I20260806 20:41:07.876209 6994 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:07.876371 6994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:07.883219 6994 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 20:41:07.886372 7033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:07.887907 6994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:41:07.888062 6994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "1c74e0674f4a4032bdeb10ac71661107"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "2555827197be919af84585ed920644e1"
server_key_iv: "7b7f21351e6671be0df767b571ec110a"
server_key_version: "encryptionkey@0"
I20260806 20:41:07.889377 6994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:07.929592 6994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:07.934530 6994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:07.934813 6994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:07.935932 6994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:07.937568 6994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:07.937697 6994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:07.937803 6994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:07.937865 6994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:07.971379 6994 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:37675
I20260806 20:41:07.971505 7149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:37675 every 8 connection(s)
I20260806 20:41:07.972846 6994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:41:07.974759 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 6994
I20260806 20:41:07.975512 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:32837
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:35589
--webserver_port=41443
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:41:07 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048867, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:08.000032 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:07.976101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:53351 (local address 127.23.200.62:35589)
0806 20:41:07.976837 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:07.976854 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:07.977439 (+ 585us) server_negotiation.cc:407] Connection header received
0806 20:41:07.980581 (+ 3142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:07.980598 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:07.980925 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:07.981223 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:07.983250 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:07.984466 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:07.986616 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:07.987738 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:07.987922 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:07.991413 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:07.991425 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:07.991445 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:07.991504 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:07.995663 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:07.996387 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:07.996397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:07.996408 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:07.996636 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:07.997069 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:07.997074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:07.997076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:07.997501 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:07.997715 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:07.998479 (+ 764us) server_negotiation.cc:299] Negotiation successful
0806 20:41:07.998865 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":281,"threads_started":1}
I20260806 20:41:08.000031 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:07.976269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:53351)
0806 20:41:07.977230 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:07.977287 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:07.980005 (+ 2718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:07.981434 (+ 1429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:07.981446 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:07.982234 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:07.983027 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:07.983056 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:07.984646 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:07.984652 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:07.986474 (+ 1822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:07.986488 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:07.987955 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:07.987960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:07.988142 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:07.988910 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:07.988932 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:07.991126 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:07.995817 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:07.995824 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:07.995833 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:07.996230 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:07.996749 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:07.996752 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:07.996754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:07.996928 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:07.997644 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:07.997651 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:07.997963 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 20:41:07.998216 (+ 253us) client_negotiation.cc:240] Negotiation successful
0806 20:41:07.998754 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":385,"threads_started":1}
I20260806 20:41:08.003149 7150 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35589
I20260806 20:41:08.003451 7150 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:08.004168 7150 heartbeater.cc:511] Master 127.23.200.62:35589 requested a full tablet report, sending...
I20260806 20:41:08.006527 6957 ts_manager.cc:194] Registered new tserver with Master: 1c74e0674f4a4032bdeb10ac71661107 (127.23.200.1:37675)
I20260806 20:41:08.008543 6957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:53351
W20260806 20:41:08.308759 7154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:08.309051 7154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:08.309149 7154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:08.316170 7154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:08.316275 7154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:08.316390 7154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:41:08.326134 7154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:32837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=41443
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:08.327440 7154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:08.328728 7154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:08.341897 7159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:08.341822 7160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:08.341789 7162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:08.342674 7154 server_base.cc:1034] running on GCE node
I20260806 20:41:08.343269 7154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:08.344177 7154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:08.345378 7154 hybrid_clock.cc:648] HybridClock initialized: now 1786048868345328 us; error 41 us; skew 500 ppm
I20260806 20:41:08.345691 7154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:08 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048868, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:08.348837 7154 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:41:08.350338 7154 webserver.cc:460] Webserver started at http://127.23.200.2:41443/ using document root <none> and password file <none>
I20260806 20:41:08.351046 7154 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:08.351202 7154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:08.356525 7154 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 20:41:08.359165 7169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:08.360455 7154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:08.360625 7154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "61051e54a4e0490c9f7385cfd3667f87"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "9a4c67319f2f4cffe2508780d5a0aae6"
server_key_iv: "3b051bdc0e69bc754161d64abe41d0a8"
server_key_version: "encryptionkey@0"
I20260806 20:41:08.361584 7154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:08.381695 7154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:08.385701 7154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:08.385982 7154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:08.386999 7154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:08.388284 7154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:08.388425 7154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:08.388569 7154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:08.388661 7154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:08.422605 7154 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:32837
I20260806 20:41:08.422693 7283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:32837 every 8 connection(s)
I20260806 20:41:08.424052 7154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:41:08.434480 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 7154
I20260806 20:41:08.435228 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:36023
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:35589
--webserver_port=36197
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:41:08 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048868, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:08.445263 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.426780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:40415 (local address 127.23.200.62:35589)
0806 20:41:08.426970 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.426975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.428219 (+ 1244us) server_negotiation.cc:407] Connection header received
0806 20:41:08.429958 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.429963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.430038 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.430130 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:08.431724 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.432543 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.434074 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.434764 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.434930 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.438486 (+ 3556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:08.438495 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:08.438503 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:08.438551 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:08.442077 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:08.442826 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:08.442834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:08.442840 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:08.443002 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:08.443403 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:08.443411 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:08.443417 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:08.443811 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:08.443972 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.444851 (+ 879us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.445023 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:41:08.446815 7286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.427349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:40415)
0806 20:41:08.428052 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:08.428095 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:08.429700 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:08.430396 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:08.430405 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:08.430968 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:08.431507 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.431524 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.432692 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.432696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.433899 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.433910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.434992 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.434997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.435189 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.435905 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:08.435931 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:08.438229 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:08.442210 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:08.442216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:08.442228 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:08.442702 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:08.443112 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:08.443115 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:08.443117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:08.443285 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:08.443932 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:08.443960 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:08.444411 (+ 451us) client_negotiation.cc:769] Sending connection context
0806 20:41:08.444751 (+ 340us) client_negotiation.cc:240] Negotiation successful
0806 20:41:08.445339 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":262,"threads_started":1}
I20260806 20:41:08.448354 7284 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35589
I20260806 20:41:08.448665 7284 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:08.449399 7284 heartbeater.cc:511] Master 127.23.200.62:35589 requested a full tablet report, sending...
I20260806 20:41:08.450881 6957 ts_manager.cc:194] Registered new tserver with Master: 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837)
I20260806 20:41:08.452121 6957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:40415
W20260806 20:41:08.764919 7287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:08.765321 7287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:08.765439 7287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:08.772492 7287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:08.772589 7287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:08.772697 7287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:41:08.781872 7287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:36023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=36197
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:08.783196 7287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:08.784456 7287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:08.797907 7295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:08.797983 7293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:08.798252 7292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:08.798893 7287 server_base.cc:1034] running on GCE node
I20260806 20:41:08.799479 7287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:08.800379 7287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:08.801581 7287 hybrid_clock.cc:648] HybridClock initialized: now 1786048868801542 us; error 30 us; skew 500 ppm
I20260806 20:41:08.801901 7287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:08 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048868, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:08.805047 7287 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:41:08.806447 7287 webserver.cc:460] Webserver started at http://127.23.200.3:36197/ using document root <none> and password file <none>
I20260806 20:41:08.807088 7287 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:08.807199 7287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:08.812420 7287 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 20:41:08.815013 7302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:08.816314 7287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:08.816491 7287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "0e97422439df4f5b89ef871fc5baa004"
format_stamp: "Formatted at 2026-08-06 20:41:06 on dist-test-slave-tm1g"
server_key: "a5d06d7f655268efb50e12732ddc02a5"
server_key_iv: "031aaf66befae33b8534a52788180622"
server_key_version: "encryptionkey@0"
I20260806 20:41:08.817490 7287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:08.829793 7287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:08.834091 7287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:08.834438 7287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:08.835393 7287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:08.836701 7287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:08.836786 7287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:08.836935 7287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:08.837069 7287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:08.869750 7287 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:36023
I20260806 20:41:08.869838 7417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:36023 every 8 connection(s)
I20260806 20:41:08.871270 7287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:41:08.872192 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 7287
Aug 06 20:41:08 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048868, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:08.887866 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.873300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54374 (local address 127.23.200.62:35589)
0806 20:41:08.873563 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.873573 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.874020 (+ 447us) server_negotiation.cc:407] Connection header received
0806 20:41:08.874477 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.874481 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.874579 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.874688 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:08.876405 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.877139 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.878816 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.879236 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.879404 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.882095 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:08.882101 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:08.882106 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:08.882146 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:08.885003 (+ 2857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:08.885670 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:08.885675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:08.885679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:08.885822 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:08.886248 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:08.886252 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:08.886254 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:08.886673 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:08.886969 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.887230 (+ 261us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.887453 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:41:08.891779 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.874255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:49365 (local address 127.23.200.62:35589)
0806 20:41:08.874992 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.874998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.876020 (+ 1022us) server_negotiation.cc:407] Connection header received
0806 20:41:08.877776 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.877782 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.877850 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.877967 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:08.879430 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.880169 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.881657 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.882113 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.882250 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.885491 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:08.885497 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:08.885503 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:08.885537 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:08.888712 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:08.889531 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:08.889536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:08.889540 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:08.889659 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:08.890138 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:08.890141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:08.890143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:08.890514 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:08.890696 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.891367 (+ 671us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.891517 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":551,"threads_started":1}
I20260806 20:41:08.892722 7422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.875012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:49365)
0806 20:41:08.875862 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:08.875905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:08.877536 (+ 1631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:08.878151 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:08.878159 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:08.878716 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:08.879238 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.879255 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.880332 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.880336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.881518 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.881529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.882357 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.882368 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.882679 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.883230 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:08.883250 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:08.885269 (+ 2019us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:08.888860 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:08.888866 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:08.888877 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:08.889408 (+ 531us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:08.889769 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:08.889774 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:08.889777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:08.890027 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:08.890674 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:08.890685 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:08.891048 (+ 363us) client_negotiation.cc:769] Sending connection context
0806 20:41:08.891266 (+ 218us) client_negotiation.cc:240] Negotiation successful
0806 20:41:08.891629 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":364,"threads_started":1}
I20260806 20:41:08.894074 7418 heartbeater.cc:347] Connected to a master server at 127.23.200.62:35589
I20260806 20:41:08.894436 7418 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:08.895170 7418 heartbeater.cc:511] Master 127.23.200.62:35589 requested a full tablet report, sending...
I20260806 20:41:08.896564 6957 ts_manager.cc:194] Registered new tserver with Master: 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
I20260806 20:41:08.897307 6957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:49365
I20260806 20:41:08.902751 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:41:08.909480 7421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.903673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54378 (local address 127.23.200.62:35589)
0806 20:41:08.903857 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.903865 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.904233 (+ 368us) server_negotiation.cc:407] Connection header received
0806 20:41:08.904483 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.904487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.904625 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.904769 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:41:08.905230 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.905913 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.907438 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.907861 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.908010 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.908471 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:41:08.908906 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:41:08.909055 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.909155 (+ 100us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.909233 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:41:08.910358 6957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54378:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:41:08.917496 6957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:41:08.951198 7428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.929302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32837 (local address 127.0.0.1:43712)
0806 20:41:08.942036 (+ 12734us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:08.942069 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:08.942228 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:08.943671 (+ 1443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:08.943675 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:08.943699 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:08.944058 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.944067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.945999 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.946004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.948584 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.948600 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.950066 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.950073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.950352 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.950416 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 20:41:08.950554 (+ 138us) client_negotiation.cc:240] Negotiation successful
0806 20:41:08.950752 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12610,"mutex_wait_us":246,"thread_start_us":12232,"threads_started":1}
I20260806 20:41:08.951138 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.929637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43712 (local address 127.23.200.2:32837)
0806 20:41:08.942904 (+ 13267us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.942915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.942953 (+ 38us) server_negotiation.cc:407] Connection header received
0806 20:41:08.943064 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.943074 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.943308 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.943524 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:08.944226 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.945837 (+ 1611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.948791 (+ 2954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.949834 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.950000 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.950205 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.950618 (+ 413us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.950780 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13062,"thread_start_us":12903,"threads_started":1}
I20260806 20:41:08.951844 7430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.928916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35240)
0806 20:41:08.942024 (+ 13108us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:08.942072 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:08.942242 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:08.943201 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:08.943209 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:08.943238 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:08.943689 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.943700 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.945743 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.945750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.948813 (+ 3063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.948830 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.950971 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.950978 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.951141 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.951186 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 20:41:08.951277 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 20:41:08.951396 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12758,"mutex_wait_us":8,"thread_start_us":14479,"threads_started":1}
I20260806 20:41:08.952306 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.929130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.23.200.1:37675)
0806 20:41:08.941774 (+ 12644us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.941784 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.942670 (+ 886us) server_negotiation.cc:407] Connection header received
0806 20:41:08.942781 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.942789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.942995 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.943202 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:08.943895 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.945596 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.948988 (+ 3392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.950475 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.950603 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.950788 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.951810 (+ 1022us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.951991 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12450,"thread_start_us":13287,"threads_started":1}
I20260806 20:41:08.954615 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.931928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:36023 (local address 127.0.0.1:60422)
0806 20:41:08.945282 (+ 13354us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:08.945316 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:08.945424 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:08.946212 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:08.946215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:08.946233 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:08.946778 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.946789 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.948650 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.948656 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.950762 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:08.950777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.953807 (+ 3030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.953813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:08.953980 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.954010 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 20:41:08.954086 (+ 76us) client_negotiation.cc:240] Negotiation successful
0806 20:41:08.954205 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13055,"mutex_wait_us":42,"thread_start_us":12893,"threads_started":1}
I20260806 20:41:08.955001 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:08.932734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.23.200.3:36023)
0806 20:41:08.944672 (+ 11938us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:08.944680 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:08.945531 (+ 851us) server_negotiation.cc:407] Connection header received
0806 20:41:08.945621 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:08.945627 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:08.945818 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:08.945995 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:08.946970 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.948521 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.950948 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:08.953307 (+ 2359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:08.953437 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:08.953658 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:08.954495 (+ 837us) server_negotiation.cc:299] Negotiation successful
0806 20:41:08.954655 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11776,"thread_start_us":11609,"threads_started":1}
I20260806 20:41:08.955523 7218 tablet_service.cc:1467] Processing CreateTablet for tablet a17f395a3fcd42c79efc30a10d6ae968 (DEFAULT_TABLE table=test-table [id=b2a88b898fdb463fa4ed56275c2b2943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:08.955574 7082 tablet_service.cc:1467] Processing CreateTablet for tablet a17f395a3fcd42c79efc30a10d6ae968 (DEFAULT_TABLE table=test-table [id=b2a88b898fdb463fa4ed56275c2b2943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:08.956964 7082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17f395a3fcd42c79efc30a10d6ae968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:08.956966 7218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17f395a3fcd42c79efc30a10d6ae968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:08.958334 7352 tablet_service.cc:1467] Processing CreateTablet for tablet a17f395a3fcd42c79efc30a10d6ae968 (DEFAULT_TABLE table=test-table [id=b2a88b898fdb463fa4ed56275c2b2943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:08.959625 7352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a17f395a3fcd42c79efc30a10d6ae968. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:08.969352 7434 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Bootstrap starting.
I20260806 20:41:08.973778 7434 tablet_bootstrap.cc:654] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:08.976331 7434 log.cc:824] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:08.978052 7436 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Bootstrap starting.
I20260806 20:41:08.980392 7435 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Bootstrap starting.
I20260806 20:41:08.984654 7435 tablet_bootstrap.cc:654] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:08.985711 7436 tablet_bootstrap.cc:654] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:08.985999 7435 log.cc:824] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:08.987105 7436 log.cc:824] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:08.991673 7434 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: No bootstrap required, opened a new log
I20260806 20:41:08.991735 7435 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: No bootstrap required, opened a new log
I20260806 20:41:08.991741 7436 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: No bootstrap required, opened a new log
I20260806 20:41:08.991993 7434 ts_tablet_manager.cc:1397] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Time spent bootstrapping tablet: real 0.023s user 0.016s sys 0.005s
I20260806 20:41:08.992002 7436 ts_tablet_manager.cc:1397] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260806 20:41:08.992020 7435 ts_tablet_manager.cc:1397] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260806 20:41:08.999831 7434 raft_consensus.cc:348] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.000222 7434 raft_consensus.cc:374] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:09.000330 7434 raft_consensus.cc:729] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c74e0674f4a4032bdeb10ac71661107, State: Initialized, Role: FOLLOWER
I20260806 20:41:09.000948 7434 consensus_queue.cc:260] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.002115 7150 heartbeater.cc:503] Master 127.23.200.62:35589 was elected leader, sending a full tablet report...
I20260806 20:41:09.002642 7434 ts_tablet_manager.cc:1428] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 20:41:09.003005 7436 raft_consensus.cc:348] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.003513 7436 raft_consensus.cc:374] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:09.003659 7436 raft_consensus.cc:729] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e97422439df4f5b89ef871fc5baa004, State: Initialized, Role: FOLLOWER
I20260806 20:41:09.003477 7435 raft_consensus.cc:348] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.003916 7435 raft_consensus.cc:374] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:09.004086 7435 raft_consensus.cc:729] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61051e54a4e0490c9f7385cfd3667f87, State: Initialized, Role: FOLLOWER
I20260806 20:41:09.004318 7436 consensus_queue.cc:260] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.004652 7435 consensus_queue.cc:260] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.005465 7418 heartbeater.cc:503] Master 127.23.200.62:35589 was elected leader, sending a full tablet report...
I20260806 20:41:09.006067 7436 ts_tablet_manager.cc:1428] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260806 20:41:09.014681 7284 heartbeater.cc:503] Master 127.23.200.62:35589 was elected leader, sending a full tablet report...
I20260806 20:41:09.015640 7435 ts_tablet_manager.cc:1428] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Time spent starting tablet: real 0.024s user 0.000s sys 0.018s
I20260806 20:41:09.063408 7440 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:09.063679 7440 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.065469 7440 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
I20260806 20:41:09.074404 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.066259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:58531 (local address 127.23.200.3:36023)
0806 20:41:09.066474 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:09.066478 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:09.067060 (+ 582us) server_negotiation.cc:407] Connection header received
0806 20:41:09.067363 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:09.067369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:09.067447 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:09.067548 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:09.068457 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.069181 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.072022 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.073438 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.073597 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.073645 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:09.073958 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 20:41:09.074097 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:41:09.074370 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.065886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:52293 (local address 127.23.200.2:32837)
0806 20:41:09.066117 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:09.066123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:09.067110 (+ 987us) server_negotiation.cc:407] Connection header received
0806 20:41:09.067363 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:09.067369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:09.067447 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:09.067548 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:09.068377 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.069370 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.072014 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.073264 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.073426 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.073528 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:09.073903 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 20:41:09.074038 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:41:09.074628 7444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.065761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:36023 (local address 127.23.200.1:58531)
0806 20:41:09.066946 (+ 1185us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.066968 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:09.067191 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:09.067580 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:09.067585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:09.067626 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:09.068267 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:09.068281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.069325 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.069330 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:09.071835 (+ 2505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:09.071850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.073633 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.073643 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:09.073791 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.073831 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 20:41:09.073921 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 20:41:09.074017 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":58,"spinlock_wait_cycles":3840,"thread_start_us":911,"threads_started":1}
I20260806 20:41:09.074918 7443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.065863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32837 (local address 127.23.200.1:52293)
0806 20:41:09.067020 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.067037 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:09.067204 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:09.067580 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:09.067585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:09.067619 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:09.068151 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:09.068167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.069522 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.069528 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:09.071838 (+ 2310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:09.071850 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.073477 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.073481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:09.073718 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.073755 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 20:41:09.073856 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 20:41:09.074007 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":75,"spinlock_wait_cycles":3072,"thread_start_us":982,"threads_started":1}
I20260806 20:41:09.075744 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17f395a3fcd42c79efc30a10d6ae968" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e97422439df4f5b89ef871fc5baa004" is_pre_election: true
I20260806 20:41:09.075968 7238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17f395a3fcd42c79efc30a10d6ae968" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61051e54a4e0490c9f7385cfd3667f87" is_pre_election: true
I20260806 20:41:09.076342 7372 raft_consensus.cc:2454] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 0.
I20260806 20:41:09.076426 7238 raft_consensus.cc:2454] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 0.
I20260806 20:41:09.077116 7038 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e97422439df4f5b89ef871fc5baa004, 1c74e0674f4a4032bdeb10ac71661107, 61051e54a4e0490c9f7385cfd3667f87; no voters:
I20260806 20:41:09.077531 7440 raft_consensus.cc:2781] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:41:09.077682 7440 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:41:09.077781 7440 raft_consensus.cc:3037] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:09.080017 7440 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.080616 7440 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 election: Requested vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
I20260806 20:41:09.081028 7238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17f395a3fcd42c79efc30a10d6ae968" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61051e54a4e0490c9f7385cfd3667f87"
I20260806 20:41:09.081112 7372 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a17f395a3fcd42c79efc30a10d6ae968" candidate_uuid: "1c74e0674f4a4032bdeb10ac71661107" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e97422439df4f5b89ef871fc5baa004"
I20260806 20:41:09.081212 7238 raft_consensus.cc:3037] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:09.081339 7372 raft_consensus.cc:3037] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:09.084007 7238 raft_consensus.cc:2454] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 1.
I20260806 20:41:09.084126 7372 raft_consensus.cc:2454] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c74e0674f4a4032bdeb10ac71661107 in term 1.
I20260806 20:41:09.084427 7038 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107, 61051e54a4e0490c9f7385cfd3667f87; no voters:
I20260806 20:41:09.084692 7440 raft_consensus.cc:2781] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:09.085120 7440 raft_consensus.cc:686] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 LEADER]: Becoming Leader. State: Replica: 1c74e0674f4a4032bdeb10ac71661107, State: Running, Role: LEADER
I20260806 20:41:09.085646 7440 consensus_queue.cc:237] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:09.089947 6955 catalog_manager.cc:5495] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c74e0674f4a4032bdeb10ac71661107 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "1c74e0674f4a4032bdeb10ac71661107" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } health_report { overall_health: UNKNOWN } } }
I20260806 20:41:09.110652 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.105107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35252 (local address 127.23.200.1:37675)
0806 20:41:09.105357 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:09.105363 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:09.105381 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:41:09.105471 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:09.105474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:09.105550 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:09.105710 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:41:09.106140 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.107008 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.108519 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.108937 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.109074 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.109569 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:41:09.110028 (+ 459us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:41:09.110193 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:09.110275 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 20:41:09.110381 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:41:09.119860 7238 raft_consensus.cc:1261] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Refusing update from remote peer 1c74e0674f4a4032bdeb10ac71661107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:09.120098 7372 raft_consensus.cc:1261] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Refusing update from remote peer 1c74e0674f4a4032bdeb10ac71661107: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:09.121173 7440 consensus_queue.cc:1035] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:09.121584 7445 consensus_queue.cc:1035] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 20:41:09.122922 7419 tablet.cc:2369] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:41:09.135742 7447 mvcc.cc:205] Tried to move back new op lower bound from 7315656167905652736 to 7315656167778594816. Current Snapshot: MvccSnapshot[applied={T|T < 7315656167905652736}]
I20260806 20:41:09.154551 7449 mvcc.cc:205] Tried to move back new op lower bound from 7315656167905652736 to 7315656167778594816. Current Snapshot: MvccSnapshot[applied={T|T < 7315656167905652736}]
I20260806 20:41:09.168145 6955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:09.168654 6955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54378:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:09.175557 6955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Sending DeleteTablet for 3 replicas of tablet a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:09.176985 7082 tablet_service.cc:1514] Processing DeleteTablet for tablet a17f395a3fcd42c79efc30a10d6ae968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:09 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35240
I20260806 20:41:09.182745 7218 tablet_service.cc:1514] Processing DeleteTablet for tablet a17f395a3fcd42c79efc30a10d6ae968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:09 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:43712
I20260806 20:41:09.191764 7352 tablet_service.cc:1514] Processing DeleteTablet for tablet a17f395a3fcd42c79efc30a10d6ae968 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:09 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:60422
I20260806 20:41:09.192863 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6994
I20260806 20:41:09.208021 7457 tablet_replica.cc:330] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: stopping tablet replica
I20260806 20:41:09.208765 7457 raft_consensus.cc:2229] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:41:09.209508 7457 raft_consensus.cc:2258] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:09.213802 7429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.194097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.23.200.1:37675)
0806 20:41:09.194370 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:09.194378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:09.194402 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:41:09.194484 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:09.194492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:09.194615 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:09.194713 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:09.196528 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.198189 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.200226 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.201148 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.201299 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.204817 (+ 3518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:09.204850 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:09.204876 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:09.204938 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:09.209754 (+ 4816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:09.210653 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:09.210666 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:09.210672 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:09.210887 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:09.211487 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:09.211495 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:09.211501 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:09.212250 (+ 749us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:09.212925 (+ 675us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:09.213103 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 20:41:09.213336 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 20:41:09.214334 7458 tablet_replica.cc:330] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: stopping tablet replica
I20260806 20:41:09.214949 7145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:09.216673 7459 tablet_replica.cc:330] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: stopping tablet replica
I20260806 20:41:09.219398 7458 raft_consensus.cc:2229] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:09.220585 7458 raft_consensus.cc:2258] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:09.223937 7458 ts_tablet_manager.cc:1905] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:09.225086 7459 raft_consensus.cc:2229] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:09.226028 7459 raft_consensus.cc:2258] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:09.232543 7459 ts_tablet_manager.cc:1905] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:09.234750 7458 ts_tablet_manager.cc:1918] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:09.234920 7458 log.cc:1196] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:09.235476 7458 ts_tablet_manager.cc:1939] T a17f395a3fcd42c79efc30a10d6ae968 P 61051e54a4e0490c9f7385cfd3667f87: Deleting consensus metadata
I20260806 20:41:09.237277 6943 catalog_manager.cc:4841] TS 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): tablet a17f395a3fcd42c79efc30a10d6ae968 (table test-table [id=b2a88b898fdb463fa4ed56275c2b2943]) successfully deleted
I20260806 20:41:09.241307 7459 ts_tablet_manager.cc:1918] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:09.241494 7459 log.cc:1196] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:09.242116 7459 ts_tablet_manager.cc:1939] T a17f395a3fcd42c79efc30a10d6ae968 P 0e97422439df4f5b89ef871fc5baa004: Deleting consensus metadata
I20260806 20:41:09.244074 6942 catalog_manager.cc:4841] TS 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): tablet a17f395a3fcd42c79efc30a10d6ae968 (table test-table [id=b2a88b898fdb463fa4ed56275c2b2943]) successfully deleted
I20260806 20:41:09.326754 7457 ts_tablet_manager.cc:1905] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:09.327770 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6994
I20260806 20:41:09.348223 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 7154
W20260806 20:41:09.348264 6940 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:37675 (error 108)
I20260806 20:41:09.350636 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.350093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35270)
0806 20:41:09.350290 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.350447 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 20:41:09.350905 6940 catalog_manager.cc:4584] TS 1c74e0674f4a4032bdeb10ac71661107 (127.23.200.1:37675): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a17f395a3fcd42c79efc30a10d6ae968: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
I20260806 20:41:09.363149 7431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.349140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.23.200.2:32837)
0806 20:41:09.349370 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:09.349378 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:09.349625 (+ 247us) server_negotiation.cc:407] Connection header received
0806 20:41:09.349703 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:09.349707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:09.349792 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:09.349908 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:09.351691 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.352527 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.353831 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.354269 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.354453 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.357109 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:09.357131 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:09.357145 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:09.357181 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:09.360170 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:09.360853 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:09.360861 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:09.360863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:09.360984 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:09.361376 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:09.361381 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:09.361386 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:09.361757 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:09.362055 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:09.362634 (+ 579us) server_negotiation.cc:299] Negotiation successful
0806 20:41:09.362855 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 20:41:09.363898 7279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:09.404584 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.404016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35276)
0806 20:41:09.404305 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.404429 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260806 20:41:09.405651 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.405284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35278)
0806 20:41:09.405452 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.405527 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 20:41:09.452314 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 7154
I20260806 20:41:09.466485 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.465874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35288)
0806 20:41:09.466131 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.466248 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 20:41:09.467634 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.467243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35304)
0806 20:41:09.467412 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.467497 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 20:41:09.471175 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 7287
I20260806 20:41:09.485579 7433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.471954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.23.200.3:36023)
0806 20:41:09.472142 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:09.472148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:09.472169 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:41:09.472291 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:09.472298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:09.472375 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:09.472474 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:09.473805 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.474958 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.476346 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:09.476971 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:09.477093 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:09.479895 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:09.479909 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:09.479926 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:09.479957 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:09.482862 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:09.483478 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:09.483485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:09.483487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:09.483663 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:09.484078 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:09.484084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:09.484087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:09.484412 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:09.484723 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:09.485033 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 20:41:09.485262 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:41:09.486199 7412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:09.548403 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.547849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35320)
0806 20:41:09.548079 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.548187 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 20:41:09.549471 7432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:09.549052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:37675 (local address 127.0.0.1:35336)
0806 20:41:09.549260 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:09.549345 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:37675: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 20:41:09.574720 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 7287
I20260806 20:41:09.593199 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 6923
I20260806 20:41:09.594398 6987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:09.707952 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 6923
I20260806 20:41:09.731135 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35589
--webserver_interface=127.23.200.62
--webserver_port=35015
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:35589
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:10.061103 7465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:10.061375 7465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:10.061486 7465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:10.068567 7465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:41:10.068677 7465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:10.068724 7465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:10.068774 7465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:41:10.068811 7465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:41:10.078622 7465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:35589
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:35589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=35015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:10.080008 7465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:10.081293 7465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:10.093108 7473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:10.093173 7470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:10.093148 7471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:10.094025 7465 server_base.cc:1034] running on GCE node
I20260806 20:41:10.094815 7465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:10.096101 7465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:10.097327 7465 hybrid_clock.cc:648] HybridClock initialized: now 1786048870097268 us; error 51 us; skew 500 ppm
I20260806 20:41:10.097702 7465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:10 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048870, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:10.105382 7465 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:41:10.106859 7465 webserver.cc:460] Webserver started at http://127.23.200.62:35015/ using document root <none> and password file <none>
I20260806 20:41:10.107544 7465 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:10.107712 7465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:10.113042 7465 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 20:41:10.115803 7481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:10.117066 7465 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:41:10.117259 7465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "5a3ff2e24f054334a20b77cc78026be4"
format_stamp: "Formatted at 2026-08-06 20:41:04 on dist-test-slave-tm1g"
server_key: "93b10fc1edb313c89cfd5d018436a09c"
server_key_iv: "f8753f65f298db887a576be0a7f3c340"
server_key_version: "encryptionkey@0"
I20260806 20:41:10.118218 7465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:10.131326 7465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:10.134912 7465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:10.135205 7465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:10.153334 7465 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:35589
I20260806 20:41:10.153366 7532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:35589 every 8 connection(s)
I20260806 20:41:10.154764 7465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:41:10.155926 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 7465
I20260806 20:41:10.156980 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:37675
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:35589
--webserver_port=33877
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:41:10.164489 7533 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:41:10.168676 7533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap starting.
I20260806 20:41:10.189268 7533 log.cc:824] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:10.205487 7533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:41:10.206032 7533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4: Bootstrap complete.
I20260806 20:41:10.215278 7533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:10.216141 7533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3ff2e24f054334a20b77cc78026be4, State: Initialized, Role: FOLLOWER
I20260806 20:41:10.216825 7533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:10.217052 7533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:41:10.217165 7533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:41:10.217381 7533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:41:10.221105 7533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:10.221635 7533 leader_election.cc:302] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a3ff2e24f054334a20b77cc78026be4; no voters:
I20260806 20:41:10.222424 7533 leader_election.cc:288] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 20:41:10.222623 7537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:41:10.223733 7537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [term 3 LEADER]: Becoming Leader. State: Replica: 5a3ff2e24f054334a20b77cc78026be4, State: Running, Role: LEADER
I20260806 20:41:10.224284 7537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } }
I20260806 20:41:10.224660 7533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:41:10.226899 7539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a3ff2e24f054334a20b77cc78026be4. Latest consensus state: current_term: 3 leader_uuid: "5a3ff2e24f054334a20b77cc78026be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } } }
I20260806 20:41:10.227404 7539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:10.228035 7538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5a3ff2e24f054334a20b77cc78026be4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a3ff2e24f054334a20b77cc78026be4" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 35589 } } }
I20260806 20:41:10.228348 7538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a3ff2e24f054334a20b77cc78026be4 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:10.234933 7543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:41:10.244107 7543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:41:10.245577 7543 catalog_manager.cc:1259] Loaded cluster ID: e44719e0728d4b5d97ce2b0fb3ed1d28
I20260806 20:41:10.245718 7543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 20:41:10.247754 7543 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f705d79e9ad google::LogMessage::Fail() at ??:0
@ 0x7f705d7a188a google::LogMessage::SendToLog() at ??:0
@ 0x7f705d79e4de google::LogMessage::Flush() at ??:0
@ 0x7f705d79fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f70731d5512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f70731d2976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f707325a610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f707325a439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f70603ea345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f70603ed91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f70603ed799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f70603c8eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f70636a26db start_thread at ??:0
@ 0x7f705c63861f clone at ??:0
W20260806 20:41:10.544488 7535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:10.544736 7535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:10.544914 7535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:10.552112 7535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:10.552230 7535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:10.552420 7535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:41:10.562161 7535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:37675
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=33877
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:10.563797 7535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:10.565699 7535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:10.580080 7576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:10.580121 7579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:10.580168 7577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:10.581108 7535 server_base.cc:1034] running on GCE node
I20260806 20:41:10.581842 7535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:10.583211 7535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:10.584422 7535 hybrid_clock.cc:648] HybridClock initialized: now 1786048870584370 us; error 38 us; skew 500 ppm
I20260806 20:41:10.584816 7535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:10 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048870, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:10.589305 7535 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:41:10.591267 7535 webserver.cc:460] Webserver started at http://127.23.200.1:33877/ using document root <none> and password file <none>
I20260806 20:41:10.592226 7535 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:10.592392 7535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:10.599383 7535 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:41:10.603235 7586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:10.604671 7535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:41:10.604820 7535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "1c74e0674f4a4032bdeb10ac71661107"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "2555827197be919af84585ed920644e1"
server_key_iv: "7b7f21351e6671be0df767b571ec110a"
server_key_version: "encryptionkey@0"
I20260806 20:41:10.605945 7535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:10.634873 7535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:10.639055 7535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:10.639328 7535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:10.640383 7535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:10.642417 7593 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 20:41:10.648003 7535 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 20:41:10.648142 7535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260806 20:41:10.648291 7535 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 20:41:10.651351 7535 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 20:41:10.651473 7535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260806 20:41:10.651773 7593 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Bootstrap starting.
I20260806 20:41:10.659694 7593 log.cc:824] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:10.674230 7593 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:41:10.675129 7593 tablet_bootstrap.cc:492] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Bootstrap complete.
I20260806 20:41:10.675745 7593 ts_tablet_manager.cc:1397] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20260806 20:41:10.686239 7593 raft_consensus.cc:348] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:10.687917 7593 raft_consensus.cc:729] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c74e0674f4a4032bdeb10ac71661107, State: Initialized, Role: FOLLOWER
I20260806 20:41:10.688545 7593 consensus_queue.cc:260] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:10.690091 7593 ts_tablet_manager.cc:1428] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260806 20:41:10.693797 7535 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:37675
I20260806 20:41:10.693818 7703 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:37675 every 8 connection(s)
I20260806 20:41:10.695014 7535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:41:10.696758 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 7535
I20260806 20:41:10.697551 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:32837
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:35589
--webserver_port=41443
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:41:10.703128 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:10.697503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:58871)
0806 20:41:10.701751 (+ 4248us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:10.702024 (+ 273us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3923,"thread_start_us":3704,"threads_started":1}
W20260806 20:41:10.703755 7704 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:35589 (0 consecutive failures): Network error: Failed to ping master at 127.23.200.62:35589: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
I20260806 20:41:10.704641 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:10.704240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:34675)
0806 20:41:10.704412 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:10.704503 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 20:41:10.705564 7706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:10.705193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:33103)
0806 20:41:10.705375 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:10.705452 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 20:41:10.705803 7704 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 20:41:11.028291 7708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:11.028556 7708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:11.028713 7708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:11.035691 7708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:11.035781 7708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:11.035929 7708 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:41:11.045143 7708 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:32837
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=41443
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:11.046372 7708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:11.047660 7708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:11.060395 7713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:11.060393 7716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:11.060451 7714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:11.061231 7708 server_base.cc:1034] running on GCE node
I20260806 20:41:11.061733 7708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:11.062616 7708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:11.063812 7708 hybrid_clock.cc:648] HybridClock initialized: now 1786048871063771 us; error 40 us; skew 500 ppm
I20260806 20:41:11.064121 7708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:11 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:11.067168 7708 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:41:11.068657 7708 webserver.cc:460] Webserver started at http://127.23.200.2:41443/ using document root <none> and password file <none>
I20260806 20:41:11.069307 7708 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:11.069458 7708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:11.074535 7708 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:41:11.077137 7723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:11.078467 7708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:11.078641 7708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "61051e54a4e0490c9f7385cfd3667f87"
format_stamp: "Formatted at 2026-08-06 20:41:05 on dist-test-slave-tm1g"
server_key: "9a4c67319f2f4cffe2508780d5a0aae6"
server_key_iv: "3b051bdc0e69bc754161d64abe41d0a8"
server_key_version: "encryptionkey@0"
I20260806 20:41:11.079595 7708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:11.093629 7708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:11.097414 7708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:11.097715 7708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:11.098716 7708 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:11.100004 7708 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:11.100128 7708 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:11.100251 7708 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:11.100342 7708 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:11.132629 7708 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:32837
I20260806 20:41:11.132712 7838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:32837 every 8 connection(s)
I20260806 20:41:11.134083 7708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:41:11.138628 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.136880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:58701)
0806 20:41:11.137589 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.137794 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":257,"threads_started":1}
W20260806 20:41:11.139451 7839 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:35589 (0 consecutive failures): Network error: Failed to ping master at 127.23.200.62:35589: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
I20260806 20:41:11.140229 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.139932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:57835)
0806 20:41:11.140068 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.140131 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260806 20:41:11.141106 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.140813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:49751)
0806 20:41:11.140922 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.140997 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260806 20:41:11.141390 7839 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 20:41:11.141938 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 7708
I20260806 20:41:11.142617 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:36023
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:35589
--webserver_port=36197
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:38115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:11.463971 7842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:11.464218 7842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:11.464309 7842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:11.471249 7842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:11.471344 7842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:11.471447 7842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:41:11.481082 7842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:36023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=36197
--tserver_master_addrs=127.23.200.62:35589
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:11.482239 7842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:11.483603 7842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:11.496336 7848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:11.496353 7847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:11.496390 7850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:11.497192 7842 server_base.cc:1034] running on GCE node
I20260806 20:41:11.497689 7842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:11.498615 7842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:11.499815 7842 hybrid_clock.cc:648] HybridClock initialized: now 1786048871499779 us; error 33 us; skew 500 ppm
I20260806 20:41:11.500116 7842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:11 dist-test-slave-tm1g krb5kdc[6358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048871, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:11.503106 7842 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:41:11.504526 7842 webserver.cc:460] Webserver started at http://127.23.200.3:36197/ using document root <none> and password file <none>
I20260806 20:41:11.505127 7842 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:11.505226 7842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:11.510210 7842 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:41:11.512802 7857 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:11.514076 7842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:41:11.514248 7842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "0e97422439df4f5b89ef871fc5baa004"
format_stamp: "Formatted at 2026-08-06 20:41:06 on dist-test-slave-tm1g"
server_key: "a5d06d7f655268efb50e12732ddc02a5"
server_key_iv: "031aaf66befae33b8534a52788180622"
server_key_version: "encryptionkey@0"
I20260806 20:41:11.515309 7842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:11.540692 7842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:11.544432 7842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:11.544718 7842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:11.545715 7842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:11.547001 7842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:11.547137 7842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:11.547259 7842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:11.547355 7842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:11.581595 7842 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:36023
I20260806 20:41:11.581689 7972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:36023 every 8 connection(s)
I20260806 20:41:11.583159 7842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:41:11.585069 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 7842
I20260806 20:41:11.588104 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.586135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:44039)
0806 20:41:11.586993 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.587278 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":400,"threads_started":1}
W20260806 20:41:11.588814 7973 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:35589 (0 consecutive failures): Network error: Failed to ping master at 127.23.200.62:35589: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
I20260806 20:41:11.589562 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.589249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:49431)
0806 20:41:11.589406 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.589467 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 20:41:11.590378 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.590011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:34393)
0806 20:41:11.590144 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.590224 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 20:41:11.590608 7973 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 20:41:11.707330 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:11.706361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:37937)
0806 20:41:11.706885 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:11.707027 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":221,"threads_started":1}
I20260806 20:41:12.142752 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.141872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:34143)
0806 20:41:12.142431 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:12.142541 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":278,"threads_started":1}
I20260806 20:41:12.409850 7979 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:12.410132 7979 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:12.411919 7979 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
Aug 06 20:41:12 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048870, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
Aug 06 20:41:12 dist-test-slave-tm1g krb5kdc[6358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048870, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:41:12.452111 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.412180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:36023 (local address 127.23.200.1:36479)
0806 20:41:12.426516 (+ 14336us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:12.426533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:12.426908 (+ 375us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:12.430407 (+ 3499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:12.430418 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:12.431567 (+ 1149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:12.432795 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:12.432822 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.435114 (+ 2292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.435119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:12.437674 (+ 2555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:12.437689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.439206 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.439213 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:12.439635 (+ 422us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:12.440719 (+ 1084us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:12.440756 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:12.443391 (+ 2635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:12.447811 (+ 4420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:12.447820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:12.447835 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:12.448594 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:12.449095 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:12.449100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:12.449104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:12.449388 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:12.450362 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:12.450370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:12.450869 (+ 499us) client_negotiation.cc:769] Sending connection context
0806 20:41:12.451224 (+ 355us) client_negotiation.cc:240] Negotiation successful
0806 20:41:12.451758 (+ 534us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14222,"mutex_wait_us":684,"spinlock_wait_cycles":115712,"thread_start_us":13642,"threads_started":1}
I20260806 20:41:12.452590 7983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.412657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:36479 (local address 127.23.200.3:36023)
0806 20:41:12.425409 (+ 12752us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:12.425434 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:12.426660 (+ 1226us) server_negotiation.cc:407] Connection header received
0806 20:41:12.429347 (+ 2687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:12.429359 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:12.429853 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:12.430216 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:12.433012 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.434909 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.437907 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.438936 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.439118 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:12.443667 (+ 4549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:12.443699 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:12.443716 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:12.443769 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:12.447660 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:12.448740 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:12.448749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:12.448756 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:12.448951 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:12.449516 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:12.449521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:12.449525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:12.450181 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:12.450376 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:12.451857 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0806 20:41:12.452164 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12475,"thread_start_us":12786,"threads_started":1}
I20260806 20:41:12.452788 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.412106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:32837 (local address 127.23.200.1:46879)
0806 20:41:12.426156 (+ 14050us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:12.426215 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:12.426907 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:12.430407 (+ 3500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:12.430418 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:12.431567 (+ 1149us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:12.432795 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:12.432822 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.435119 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.435123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:12.437674 (+ 2551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:12.437689 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.439193 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.439200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:12.439633 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:12.440890 (+ 1257us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:12.440922 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:12.443618 (+ 2696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:12.447811 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:12.447820 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:12.447835 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:12.448592 (+ 757us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:12.449095 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:12.449100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:12.449104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:12.449388 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:12.450362 (+ 974us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:12.450370 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:12.450869 (+ 499us) client_negotiation.cc:769] Sending connection context
0806 20:41:12.451224 (+ 355us) client_negotiation.cc:240] Negotiation successful
0806 20:41:12.452417 (+ 1193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13867,"mutex_wait_us":1245,"spinlock_wait_cycles":85760,"thread_start_us":14603,"threads_started":1}
I20260806 20:41:12.453841 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.412330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:46879 (local address 127.23.200.2:32837)
0806 20:41:12.426623 (+ 14293us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:12.426638 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:12.426687 (+ 49us) server_negotiation.cc:407] Connection header received
0806 20:41:12.429396 (+ 2709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:12.429405 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:12.429853 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:12.430216 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:12.433012 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.434909 (+ 1897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.437907 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:12.438936 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:12.439118 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:12.443874 (+ 4756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:12.443886 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:12.443901 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:12.443948 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:12.447660 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:12.448740 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:12.448749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:12.448756 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:12.448977 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:12.449516 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:12.449521 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:12.449525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:12.450178 (+ 653us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:12.450382 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:12.453071 (+ 2689us) server_negotiation.cc:299] Negotiation successful
0806 20:41:12.453457 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14092,"thread_start_us":13907,"threads_started":1}
W20260806 20:41:12.455626 7589 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
W20260806 20:41:12.456149 7590 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:12.456297 7590 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107; no voters: 0e97422439df4f5b89ef871fc5baa004, 61051e54a4e0490c9f7385cfd3667f87
I20260806 20:41:12.456722 7979 raft_consensus.cc:2726] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:41:12.591879 7984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.591069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:40135)
0806 20:41:12.591493 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:12.591629 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":199,"threads_started":1}
I20260806 20:41:12.708709 7981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:12.708330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:56847)
0806 20:41:12.708486 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:12.708581 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 20:41:13.144236 7985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:13.143590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:40295)
0806 20:41:13.144006 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:13.144085 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
I20260806 20:41:13.593302 7986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:13.592673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:42827)
0806 20:41:13.593086 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:13.593162 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":232,"threads_started":1}
I20260806 20:41:13.710536 7987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:13.709641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:59319)
0806 20:41:13.710100 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:13.710194 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":245,"threads_started":1}
I20260806 20:41:14.145823 7988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:14.145149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:33847)
0806 20:41:14.145591 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:14.145670 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":260,"threads_started":1}
I20260806 20:41:14.522398 7989 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:14.522586 7989 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:14.523211 7989 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
W20260806 20:41:14.524242 7590 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
W20260806 20:41:14.524432 7589 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:14.524551 7589 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107; no voters: 0e97422439df4f5b89ef871fc5baa004, 61051e54a4e0490c9f7385cfd3667f87
I20260806 20:41:14.524766 7989 raft_consensus.cc:2726] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:41:14.595016 7990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:14.594236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:45375)
0806 20:41:14.594784 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:14.594862 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":345,"threads_started":1}
I20260806 20:41:14.712152 7991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:14.711454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:57241)
0806 20:41:14.711868 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:14.711971 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":201,"threads_started":1}
I20260806 20:41:15.147379 7992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:15.146712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:59465)
0806 20:41:15.147131 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:15.147213 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":209,"threads_started":1}
I20260806 20:41:15.596495 7993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:15.595777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:60955)
0806 20:41:15.596193 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:15.596289 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":186,"threads_started":1}
I20260806 20:41:15.713828 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:15.713105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:50859)
0806 20:41:15.713559 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:15.713652 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":216,"threads_started":1}
I20260806 20:41:16.148963 7995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:16.148253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:56359)
0806 20:41:16.148729 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:16.148807 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":280,"threads_started":1}
I20260806 20:41:16.557447 7997 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:16.557734 7997 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:16.558547 7997 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
W20260806 20:41:16.559427 7589 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
W20260806 20:41:16.559651 7590 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:16.559815 7590 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107; no voters: 0e97422439df4f5b89ef871fc5baa004, 61051e54a4e0490c9f7385cfd3667f87
I20260806 20:41:16.560065 7997 raft_consensus.cc:2726] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:41:16.598029 7998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:16.597313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:36247)
0806 20:41:16.597803 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:16.597881 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":303,"threads_started":1}
I20260806 20:41:16.715359 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:16.714650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:53661)
0806 20:41:16.715085 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:16.715182 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":230,"threads_started":1}
I20260806 20:41:17.150600 8000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:17.149843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:45585)
0806 20:41:17.150276 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:17.150426 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260806 20:41:17.599613 8001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:17.598879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:36943)
0806 20:41:17.599358 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:17.599439 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":228,"threads_started":1}
I20260806 20:41:17.716885 8002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:17.716190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:38773)
0806 20:41:17.716611 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:17.716689 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":223,"threads_started":1}
I20260806 20:41:18.152433 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:18.151471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:53085)
0806 20:41:18.152102 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:18.152239 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":311,"threads_started":1}
I20260806 20:41:18.601289 8004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:18.600511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:43949)
0806 20:41:18.601023 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:18.601110 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":233,"threads_started":1}
I20260806 20:41:18.718550 8005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:18.717774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:50577)
0806 20:41:18.718233 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:18.718389 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":203,"threads_started":1}
I20260806 20:41:19.154201 8006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:19.153317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:48113)
0806 20:41:19.153852 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:19.153955 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":259,"threads_started":1}
I20260806 20:41:19.602777 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:19.602094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:58357)
0806 20:41:19.602529 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:19.602605 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":253,"threads_started":1}
I20260806 20:41:19.720163 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:19.719364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:32817)
0806 20:41:19.719908 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:19.719989 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":282,"threads_started":1}
I20260806 20:41:20.155970 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:20.155181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:52989)
0806 20:41:20.155703 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:20.155801 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":298,"threads_started":1}
I20260806 20:41:20.239032 8010 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:20.239243 8010 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:20.239867 8010 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
W20260806 20:41:20.240916 7590 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
W20260806 20:41:20.241123 7589 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:20.241221 7589 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107; no voters: 0e97422439df4f5b89ef871fc5baa004, 61051e54a4e0490c9f7385cfd3667f87
I20260806 20:41:20.241452 8010 raft_consensus.cc:2726] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:41:20.604220 8011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:20.603560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:54157)
0806 20:41:20.603981 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:20.604070 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":199,"threads_started":1}
I20260806 20:41:20.721918 8012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:20.721060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:35531)
0806 20:41:20.721611 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:20.721719 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":296,"threads_started":1}
I20260806 20:41:21.157606 8013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:21.156899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:48503)
0806 20:41:21.157344 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:21.157434 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":225,"threads_started":1}
I20260806 20:41:21.605738 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:21.605060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:39071)
0806 20:41:21.605489 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:21.605565 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
I20260806 20:41:21.723493 8015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:21.722793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:50963)
0806 20:41:21.723245 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:21.723322 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":238,"threads_started":1}
I20260806 20:41:22.159260 8016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:22.158559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:51843)
0806 20:41:22.159020 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:22.159099 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":236,"threads_started":1}
I20260806 20:41:22.319828 8017 raft_consensus.cc:482] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:22.320009 8017 raft_consensus.cc:504] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c74e0674f4a4032bdeb10ac71661107" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 37675 } } peers { permanent_uuid: "61051e54a4e0490c9f7385cfd3667f87" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 32837 } } peers { permanent_uuid: "0e97422439df4f5b89ef871fc5baa004" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 36023 } }
I20260806 20:41:22.320586 8017 leader_election.cc:288] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837), 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023)
W20260806 20:41:22.321657 7590 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 61051e54a4e0490c9f7385cfd3667f87 (127.23.200.2:32837): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
W20260806 20:41:22.321902 7589 leader_election.cc:341] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0e97422439df4f5b89ef871fc5baa004 (127.23.200.3:36023): Not found: Tablet not found: a17f395a3fcd42c79efc30a10d6ae968
I20260806 20:41:22.322049 7589 leader_election.cc:302] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1c74e0674f4a4032bdeb10ac71661107; no voters: 0e97422439df4f5b89ef871fc5baa004, 61051e54a4e0490c9f7385cfd3667f87
I20260806 20:41:22.322355 8017 raft_consensus.cc:2726] T a17f395a3fcd42c79efc30a10d6ae968 P 1c74e0674f4a4032bdeb10ac71661107 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:41:22.607378 8018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:22.606631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:45661)
0806 20:41:22.607124 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:22.607212 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":249,"threads_started":1}
I20260806 20:41:22.725107 8019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:22.724277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:45237)
0806 20:41:22.724812 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:22.724897 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":306,"threads_started":1}
I20260806 20:41:23.160883 8020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:23.160102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:41967)
0806 20:41:23.160631 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:23.160722 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":314,"threads_started":1}
I20260806 20:41:23.609037 8021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:23.608344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:41665)
0806 20:41:23.608788 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:23.608865 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":264,"threads_started":1}
I20260806 20:41:23.726742 8022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:23.725967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:58951)
0806 20:41:23.726484 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:23.726566 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":249,"threads_started":1}
I20260806 20:41:24.162418 8023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:24.161736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:46667)
0806 20:41:24.162153 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:24.162245 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":216,"threads_started":1}
I20260806 20:41:24.610745 8024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:24.609976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:41817)
0806 20:41:24.610477 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:24.610578 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":236,"threads_started":1}
I20260806 20:41:24.728325 8025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:24.727642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:50987)
0806 20:41:24.728059 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:24.728139 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":213,"threads_started":1}
I20260806 20:41:25.164244 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:25.163374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:53551)
0806 20:41:25.163907 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:25.164032 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":296,"threads_started":1}
I20260806 20:41:25.612637 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:25.611796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:48943)
0806 20:41:25.612333 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:25.612422 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":322,"threads_started":1}
I20260806 20:41:25.730108 8028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:25.729426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.1:37607)
0806 20:41:25.729848 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:25.729926 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":210,"threads_started":1}
I20260806 20:41:26.166206 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:26.165487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.2:59443)
0806 20:41:26.165941 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:26.166041 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":231,"threads_started":1}
I20260806 20:41:26.595633 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 7535
I20260806 20:41:26.614478 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:26.613640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:35589 (local address 127.23.200.3:37905)
0806 20:41:26.614144 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:26.614249 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:35589: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":273,"threads_started":1}
I20260806 20:41:26.615391 8030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:26.596784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.23.200.1:37675)
0806 20:41:26.597319 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:26.597327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:26.597351 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:41:26.597411 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:26.597417 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:26.597639 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:26.597810 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:26.599391 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:26.600421 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:26.601675 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:26.602276 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:26.602464 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:26.609625 (+ 7161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:26.609636 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:26.609650 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:26.609689 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:26.612555 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:26.613308 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:26.613315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:26.613318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:26.613454 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:26.613859 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:26.613864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:26.613867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:26.614238 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:26.614608 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:26.614926 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 20:41:26.615153 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":255,"threads_started":1}
I20260806 20:41:26.616639 7699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:26.706120 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 7535
I20260806 20:41:26.726768 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 7708
I20260806 20:41:26.740085 8033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:26.727577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37064 (local address 127.23.200.2:32837)
0806 20:41:26.728104 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:26.728108 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:26.728124 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:41:26.728179 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:26.728183 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:26.728243 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:26.728334 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:26.729744 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:26.730604 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:26.731824 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:26.732248 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:26.732338 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:26.735238 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:26.735242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:26.735246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:26.735276 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:26.737706 (+ 2430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:26.738373 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:26.738378 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:26.738382 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:26.738532 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:26.738861 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:26.738864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:26.738865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:26.739120 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:26.739333 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:26.739543 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 20:41:26.739773 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
I20260806 20:41:26.740785 7834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:26.825989 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 7708
I20260806 20:41:26.844148 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 7842
I20260806 20:41:26.857460 8035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:26.845025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.23.200.3:36023)
0806 20:41:26.845654 (+ 629us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:26.845658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:26.845676 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:41:26.845731 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:26.845735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:26.845803 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:26.845893 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:26.847389 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:26.848177 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:26.849429 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:26.849891 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:26.849991 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:26.852433 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:26.852437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:26.852441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:26.852484 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:26.855221 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:26.855798 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:26.855802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:26.855804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:26.855942 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:26.856252 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:26.856255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:26.856256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:26.856501 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:26.856712 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:26.856887 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 20:41:26.857047 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":352,"threads_started":1}
I20260806 20:41:26.858165 7968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:26.943506 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 7842
2026-08-06T20:41:26Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24089 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:41:27 dist-test-slave-tm1g krb5kdc[8039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:41:27 dist-test-slave-tm1g krb5kdc[8039](info): set up 2 sockets
Aug 06 20:41:27 dist-test-slave-tm1g krb5kdc[8039](info): commencing operation
krb5kdc: starting...
W20260806 20:41:28.072680 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:41:28 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:41:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:41:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:41:28.313297 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45869
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:28.643857 8052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:28.644198 8052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:28.644357 8052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:28.651489 8052 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 20:41:28.651607 8052 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 20:41:28.651693 8052 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 20:41:28.651793 8052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:41:28.651895 8052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:28.652019 8052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:28.652107 8052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:41:28.652190 8052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:41:28.661654 8052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:28.663168 8052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:28.664474 8052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:28.675956 8057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:28.676002 8058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:28.676043 8060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:28.677165 8052 server_base.cc:1034] running on GCE node
I20260806 20:41:28.677865 8052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:28.679239 8052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:28.680490 8052 hybrid_clock.cc:648] HybridClock initialized: now 1786048888680431 us; error 34 us; skew 500 ppm
I20260806 20:41:28.680816 8052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:28 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048888, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:28.684008 8052 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:41:28.685532 8052 webserver.cc:460] Webserver started at http://127.23.200.62:44493/ using document root <none> and password file <none>
I20260806 20:41:28.686218 8052 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:28.686352 8052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:28.686640 8052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:28.689450 8052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "fd1d4d6e9f5249928d329c78a7464f02"
format_stamp: "Formatted at 2026-08-06 20:41:28 on dist-test-slave-tm1g"
server_key: "1434dc336f9d4821256cea2bf90e3bc7"
server_key_iv: "f47ab5fa85581688763ab1eef6612b84"
server_key_version: "encryptionkey@0"
I20260806 20:41:28.690164 8052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "fd1d4d6e9f5249928d329c78a7464f02"
format_stamp: "Formatted at 2026-08-06 20:41:28 on dist-test-slave-tm1g"
server_key: "1434dc336f9d4821256cea2bf90e3bc7"
server_key_iv: "f47ab5fa85581688763ab1eef6612b84"
server_key_version: "encryptionkey@0"
I20260806 20:41:28.695009 8052 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:41:28.698385 8067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:28.699715 8052 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:41:28.699852 8052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "fd1d4d6e9f5249928d329c78a7464f02"
format_stamp: "Formatted at 2026-08-06 20:41:28 on dist-test-slave-tm1g"
server_key: "1434dc336f9d4821256cea2bf90e3bc7"
server_key_iv: "f47ab5fa85581688763ab1eef6612b84"
server_key_version: "encryptionkey@0"
I20260806 20:41:28.700029 8052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:28.733317 8052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:28.737752 8052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:28.737984 8052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:28.756688 8052 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:45869
I20260806 20:41:28.756709 8119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:45869 every 8 connection(s)
I20260806 20:41:28.758019 8052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:41:28.761819 8120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:28.766707 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8052
I20260806 20:41:28.766974 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:41:28.767797 24352 external_mini_cluster.cc:1428] Setting key 3e1ef61945b7620b0f46c001d32411ed
I20260806 20:41:28.773433 8120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap starting.
I20260806 20:41:28.777160 8120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:28.778409 8120 log.cc:824] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:28.781805 8120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: No bootstrap required, opened a new log
Aug 06 20:41:28 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:28.790062 8120 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:28.790474 8120 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:28.790597 8120 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d4d6e9f5249928d329c78a7464f02, State: Initialized, Role: FOLLOWER
I20260806 20:41:28.791225 8120 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:28.791508 8120 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:41:28.791671 8120 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:41:28.791850 8120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:28.792148 8123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:28.769914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.23.200.62:45869)
0806 20:41:28.770802 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:28.770814 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:28.770857 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:41:28.772567 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:28.772589 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:28.772991 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:28.773303 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:28.775242 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:28.776545 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:28.778586 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:28.779669 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:28.779843 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:28.784676 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:28.784684 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:28.784696 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:28.784735 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:28.787864 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:28.788565 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:28.788571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:28.788577 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:28.788751 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:28.789185 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:28.789194 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:28.789196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:28.789706 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:28.790018 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:28.790526 (+ 508us) server_negotiation.cc:299] Negotiation successful
0806 20:41:28.790967 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":418,"threads_started":1}
I20260806 20:41:28.794706 8120 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:28.795238 8120 leader_election.cc:302] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1d4d6e9f5249928d329c78a7464f02; no voters:
I20260806 20:41:28.795779 8120 leader_election.cc:288] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:41:28.795938 8125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:28.797152 8125 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 LEADER]: Becoming Leader. State: Replica: fd1d4d6e9f5249928d329c78a7464f02, State: Running, Role: LEADER
I20260806 20:41:28.797690 8125 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:28.797943 8120 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:41:28.800652 8126 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd1d4d6e9f5249928d329c78a7464f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } } }
I20260806 20:41:28.801132 8126 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:28.801054 8127 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1d4d6e9f5249928d329c78a7464f02. Latest consensus state: current_term: 1 leader_uuid: "fd1d4d6e9f5249928d329c78a7464f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } } }
I20260806 20:41:28.801322 8127 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:28.803012 8134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:41:28.807299 8134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:41:28.815655 8134 catalog_manager.cc:1347] Generated new cluster ID: 8cf08da2847a45959ac98fc46c8102fc
I20260806 20:41:28.815763 8134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:41:28.835481 8134 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:41:28.837352 8134 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:41:28.844682 8134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Generated new TSK 0
I20260806 20:41:28.845471 8134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:41:28.906016 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45869
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:29.236320 8146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:29.236644 8146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:29.236811 8146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:29.243911 8146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:29.244051 8146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:29.244239 8146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:41:29.253753 8146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:29.255105 8146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:29.256397 8146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:29.269289 8151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:29.269325 8152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:29.269379 8154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:29.270397 8146 server_base.cc:1034] running on GCE node
I20260806 20:41:29.271030 8146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:29.272030 8146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:29.273224 8146 hybrid_clock.cc:648] HybridClock initialized: now 1786048889273180 us; error 45 us; skew 500 ppm
I20260806 20:41:29.273494 8146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:29 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:29.276566 8146 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:41:29.278012 8146 webserver.cc:460] Webserver started at http://127.23.200.1:41157/ using document root <none> and password file <none>
I20260806 20:41:29.278759 8146 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:29.278908 8146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:29.279210 8146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:29.282034 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "92d2365e9b614ee6b95cf8716741a4a5"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "5f180a46d5c2e2a79cbe547017116462"
server_key_iv: "60f37531562cb345afe32f27027941e8"
server_key_version: "encryptionkey@0"
I20260806 20:41:29.282845 8146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "92d2365e9b614ee6b95cf8716741a4a5"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "5f180a46d5c2e2a79cbe547017116462"
server_key_iv: "60f37531562cb345afe32f27027941e8"
server_key_version: "encryptionkey@0"
I20260806 20:41:29.287822 8146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:41:29.291424 8161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:29.292747 8146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:29.292925 8146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "92d2365e9b614ee6b95cf8716741a4a5"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "5f180a46d5c2e2a79cbe547017116462"
server_key_iv: "60f37531562cb345afe32f27027941e8"
server_key_version: "encryptionkey@0"
I20260806 20:41:29.293126 8146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:29.304225 8146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:29.308494 8146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:29.308787 8146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:29.309693 8146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:29.311031 8146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:29.311141 8146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:29.311295 8146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:29.311368 8146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:29.345434 8146 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:39649
I20260806 20:41:29.345497 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:39649 every 8 connection(s)
I20260806 20:41:29.346902 8146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:41:29.352363 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8146
I20260806 20:41:29.352578 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:41:29.353241 24352 external_mini_cluster.cc:1428] Setting key 7532206cffe8c88db6947e5a3d3b4e48
Aug 06 20:41:29 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:29.366833 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:29.349432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38931 (local address 127.23.200.62:45869)
0806 20:41:29.349905 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:29.349909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:29.350778 (+ 869us) server_negotiation.cc:407] Connection header received
0806 20:41:29.352668 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:29.352673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:29.352766 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:29.352885 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:29.354641 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.355613 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.357151 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.357555 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.357694 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:29.361075 (+ 3381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:29.361080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:29.361083 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:29.361113 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:29.363798 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:29.364529 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:29.364533 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:29.364536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:29.364651 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:29.365087 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:29.365092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:29.365097 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:29.365360 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:29.365525 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:29.366205 (+ 680us) server_negotiation.cc:299] Negotiation successful
0806 20:41:29.366455 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":271,"threads_started":1}
I20260806 20:41:29.367733 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:29.349881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:38931)
0806 20:41:29.350620 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:29.350657 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:29.352363 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:29.353125 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:29.353133 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:29.353704 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:29.354473 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:29.354493 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.355785 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.355789 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:29.357003 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:29.357015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.357789 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.357798 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:29.358039 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:29.358737 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:29.358759 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:29.360856 (+ 2097us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:29.363941 (+ 3085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:29.363950 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:29.363972 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:29.364386 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:29.364762 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:29.364768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:29.364771 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:29.364967 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:29.365500 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:29.365508 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:29.365825 (+ 317us) client_negotiation.cc:769] Sending connection context
0806 20:41:29.366090 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 20:41:29.366634 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":267,"threads_started":1}
I20260806 20:41:29.369314 8277 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45869
I20260806 20:41:29.369629 8277 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:29.370343 8277 heartbeater.cc:511] Master 127.23.200.62:45869 requested a full tablet report, sending...
I20260806 20:41:29.372584 8084 ts_manager.cc:194] Registered new tserver with Master: 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1:39649)
I20260806 20:41:29.374895 8084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:38931
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:41:29.405378 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45869
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:29.727399 8284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:29.727710 8284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:29.727875 8284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:29.735613 8284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:29.735773 8284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:29.735975 8284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:41:29.746902 8284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:29.748456 8284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:29.749778 8284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:29.763900 8290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:29.763967 8289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:29.764001 8292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:29.764683 8284 server_base.cc:1034] running on GCE node
I20260806 20:41:29.765228 8284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:29.766180 8284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:29.767407 8284 hybrid_clock.cc:648] HybridClock initialized: now 1786048889767356 us; error 47 us; skew 500 ppm
I20260806 20:41:29.767729 8284 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:29 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:29.770823 8284 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:41:29.774099 8284 webserver.cc:460] Webserver started at http://127.23.200.2:41389/ using document root <none> and password file <none>
I20260806 20:41:29.774835 8284 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:29.774968 8284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:29.775286 8284 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:29.778199 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "315c5a139a3c4b718a9826c98f83cbdf"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "b4de5d11143a868355ce652d3b4fecdb"
server_key_iv: "fae544854574a8d446d08bfca2ec0ca0"
server_key_version: "encryptionkey@0"
I20260806 20:41:29.778985 8284 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "315c5a139a3c4b718a9826c98f83cbdf"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "b4de5d11143a868355ce652d3b4fecdb"
server_key_iv: "fae544854574a8d446d08bfca2ec0ca0"
server_key_version: "encryptionkey@0"
I20260806 20:41:29.783970 8284 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:41:29.787348 8301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:29.788594 8284 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:29.788748 8284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "315c5a139a3c4b718a9826c98f83cbdf"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "b4de5d11143a868355ce652d3b4fecdb"
server_key_iv: "fae544854574a8d446d08bfca2ec0ca0"
server_key_version: "encryptionkey@0"
I20260806 20:41:29.788936 8284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:29.802526 8284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:29.806850 8284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:29.807130 8284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:29.808017 8284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:29.809270 8284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:29.809373 8284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:29.809522 8284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:29.809597 8284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:29.842490 8284 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33059
I20260806 20:41:29.842537 8417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33059 every 8 connection(s)
I20260806 20:41:29.843758 8284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:41:29.847107 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8284
I20260806 20:41:29.847352 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:41:29.847922 24352 external_mini_cluster.cc:1428] Setting key 9ef4773b3e10aca97fe44f071165c6f1
Aug 06 20:41:29 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048889, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:29.864784 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:29.846378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:45085 (local address 127.23.200.62:45869)
0806 20:41:29.846596 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:29.846602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:29.847889 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 20:41:29.849894 (+ 2005us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:29.849900 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:29.849981 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:29.850081 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:29.851546 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.852692 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.854174 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.854973 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.855118 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:29.858714 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:29.858723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:29.858728 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:29.858773 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:29.861514 (+ 2741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:29.862500 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:29.862504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:29.862506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:29.862625 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:29.863092 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:29.863095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:29.863097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:29.863331 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:29.863447 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:29.864304 (+ 857us) server_negotiation.cc:299] Negotiation successful
0806 20:41:29.864528 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 20:41:29.866231 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:29.846900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:45085)
0806 20:41:29.847733 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:29.847775 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:29.849559 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:29.850266 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:29.850275 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:29.850833 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:29.851370 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:29.851387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.852839 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.852842 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:29.854027 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:29.854039 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:29.855172 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:29.855177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:29.855369 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:29.856054 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:29.856075 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:29.858405 (+ 2330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:29.861659 (+ 3254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:29.861670 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:29.861687 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:29.862354 (+ 667us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:29.862733 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:29.862740 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:29.862746 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:29.862984 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:29.863456 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:29.863465 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:29.863872 (+ 407us) client_negotiation.cc:769] Sending connection context
0806 20:41:29.864157 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 20:41:29.864705 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":341,"threads_started":1}
I20260806 20:41:29.867789 8418 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45869
I20260806 20:41:29.868106 8418 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:29.868810 8418 heartbeater.cc:511] Master 127.23.200.62:45869 requested a full tablet report, sending...
I20260806 20:41:29.870384 8084 ts_manager.cc:194] Registered new tserver with Master: 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059)
I20260806 20:41:29.871284 8084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:45085
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:41:29.905798 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45869
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:30.231905 8423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:30.232209 8423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:30.232381 8423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:30.239359 8423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:30.239521 8423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:30.239701 8423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:41:30.249166 8423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:30.250471 8423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:30.251748 8423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:30.265233 8431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:30.265302 8428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:30.265329 8429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:30.266090 8423 server_base.cc:1034] running on GCE node
I20260806 20:41:30.266655 8423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:30.267467 8423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:30.268656 8423 hybrid_clock.cc:648] HybridClock initialized: now 1786048890268602 us; error 48 us; skew 500 ppm
I20260806 20:41:30.268954 8423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:30 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:30.272111 8423 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:41:30.273506 8423 webserver.cc:460] Webserver started at http://127.23.200.3:36605/ using document root <none> and password file <none>
I20260806 20:41:30.274173 8423 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:30.274374 8423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:30.274708 8423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:41:30.277560 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
format_stamp: "Formatted at 2026-08-06 20:41:30 on dist-test-slave-tm1g"
server_key: "95d9305eab4ee0ae468cdf6ee625a839"
server_key_iv: "864adff332f253f1a47b5cd7d98f6178"
server_key_version: "encryptionkey@0"
I20260806 20:41:30.278363 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
format_stamp: "Formatted at 2026-08-06 20:41:30 on dist-test-slave-tm1g"
server_key: "95d9305eab4ee0ae468cdf6ee625a839"
server_key_iv: "864adff332f253f1a47b5cd7d98f6178"
server_key_version: "encryptionkey@0"
I20260806 20:41:30.283344 8423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:41:30.286755 8438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:30.288139 8423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 20:41:30.288317 8423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
format_stamp: "Formatted at 2026-08-06 20:41:30 on dist-test-slave-tm1g"
server_key: "95d9305eab4ee0ae468cdf6ee625a839"
server_key_iv: "864adff332f253f1a47b5cd7d98f6178"
server_key_version: "encryptionkey@0"
I20260806 20:41:30.288524 8423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:30.302814 8423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:30.307075 8423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:30.307382 8423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:30.308305 8423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:30.309573 8423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:30.309677 8423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:30.309829 8423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:30.309907 8423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:30.343144 8423 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37317
I20260806 20:41:30.343194 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37317 every 8 connection(s)
I20260806 20:41:30.344487 8423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:41:30.347323 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8423
I20260806 20:41:30.347533 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:41:30.348097 24352 external_mini_cluster.cc:1428] Setting key bff31a748164ca846ca6f544cc0f8213
Aug 06 20:41:30 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048890, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:30.365533 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.348433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:50469 (local address 127.23.200.62:45869)
0806 20:41:30.348668 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.348675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.349725 (+ 1050us) server_negotiation.cc:407] Connection header received
0806 20:41:30.352114 (+ 2389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.352119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.352235 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.352344 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:30.353785 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.354597 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.356110 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.356558 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.356687 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.360058 (+ 3371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:30.360062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:30.360066 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:30.360096 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:30.362583 (+ 2487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.363262 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.363267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.363271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.363411 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.363833 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.363838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.363842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.364112 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:30.364268 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.364932 (+ 664us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.365169 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 20:41:30.366441 8557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.347879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:50469)
0806 20:41:30.349574 (+ 1695us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:30.349618 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:30.351862 (+ 2244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:30.352583 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:30.352591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:30.353088 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:30.353623 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.353640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.354762 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.354767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.355968 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.355980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.356788 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.356795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.357028 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.357658 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:30.357679 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:30.359852 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:30.362716 (+ 2864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:30.362724 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:30.362738 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:30.363140 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:30.363532 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:30.363537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:30.363541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:30.363720 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:30.364241 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:30.364248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:30.364568 (+ 320us) client_negotiation.cc:769] Sending connection context
0806 20:41:30.364823 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 20:41:30.365337 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1404,"thread_start_us":1166,"threads_started":1}
I20260806 20:41:30.367784 8555 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45869
I20260806 20:41:30.368031 8555 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:30.368700 8555 heartbeater.cc:511] Master 127.23.200.62:45869 requested a full tablet report, sending...
I20260806 20:41:30.370002 8084 ts_manager.cc:194] Registered new tserver with Master: 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
I20260806 20:41:30.370886 8084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:50469
I20260806 20:41:30.375995 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:41:30.377866 8277 heartbeater.cc:503] Master 127.23.200.62:45869 was elected leader, sending a full tablet report...
I20260806 20:41:30.393332 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.380773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.23.200.62:45869)
0806 20:41:30.380979 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.380988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.381363 (+ 375us) server_negotiation.cc:407] Connection header received
0806 20:41:30.381489 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.381493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.381549 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.381623 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:30.383127 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.383807 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.385204 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.385614 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.385704 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.388184 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:30.388189 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:30.388192 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:30.388221 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:30.390655 (+ 2434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.391300 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.391303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.391306 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.391483 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.391864 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.391868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.391870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.392280 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:30.392480 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.392858 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.393029 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 20:41:30.397598 8084 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:41:30.405145 8084 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:41:30.429955 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.418046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59310 (local address 127.23.200.1:39649)
0806 20:41:30.419589 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.419602 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.421309 (+ 1707us) server_negotiation.cc:407] Connection header received
0806 20:41:30.421627 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.421636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.421900 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.422069 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:30.423077 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.424299 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.427636 (+ 3337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.428659 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.428840 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.429092 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.429429 (+ 337us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.429608 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1339,"thread_start_us":1185,"threads_started":1}
I20260806 20:41:30.430091 8567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.417096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33059 (local address 127.0.0.1:44422)
0806 20:41:30.417986 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:30.418026 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:30.418200 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:30.422145 (+ 3945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:30.422155 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:30.422186 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:30.422705 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.422726 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.424778 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.424784 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.427320 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.427334 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.429160 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.429167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.429339 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.429374 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:41:30.429436 (+ 62us) client_negotiation.cc:240] Negotiation successful
0806 20:41:30.429641 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":56,"thread_start_us":1308,"threads_started":1}
I20260806 20:41:30.430922 8568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.417309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:59310)
0806 20:41:30.421185 (+ 3876us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:30.421207 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:30.421420 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:30.422410 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:30.422413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:30.422434 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:30.422863 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.422877 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.424436 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.424443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.427467 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.427482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.428879 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.428885 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.429094 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.429161 (+ 67us) client_negotiation.cc:769] Sending connection context
0806 20:41:30.430538 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0806 20:41:30.430651 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3742,"mutex_wait_us":1,"thread_start_us":3599,"threads_started":1}
I20260806 20:41:30.431025 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.419835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.23.200.3:37317)
0806 20:41:30.420713 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.420723 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.422014 (+ 1291us) server_negotiation.cc:407] Connection header received
0806 20:41:30.422338 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.422346 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.422580 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.422710 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:30.423577 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.425109 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.427840 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.429263 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.429380 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.429599 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.430526 (+ 927us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.430685 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":492,"threads_started":1}
I20260806 20:41:30.431071 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.417288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44422 (local address 127.23.200.2:33059)
0806 20:41:30.421513 (+ 4225us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.421523 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.421549 (+ 26us) server_negotiation.cc:407] Connection header received
0806 20:41:30.421632 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.421637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.421889 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.422947 (+ 1058us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:30.423159 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.424622 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.427495 (+ 2873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.428915 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.430331 (+ 1416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.430552 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.430612 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.430760 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4054,"thread_start_us":3922,"threads_started":1}
I20260806 20:41:30.431661 8571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.419450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37317 (local address 127.0.0.1:38204)
0806 20:41:30.421899 (+ 2449us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:30.421918 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:30.422113 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:30.422941 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:30.422944 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:30.422962 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:30.423389 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.423401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.425289 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.425312 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.427679 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.427694 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.429748 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.429758 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.429929 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.429977 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 20:41:30.430085 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 20:41:30.431383 (+ 1298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2320,"thread_start_us":2171,"threads_started":1}
I20260806 20:41:30.434576 8351 tablet_service.cc:1467] Processing CreateTablet for tablet 5612384b173a48fd957e161393a9db5a (DEFAULT_TABLE table=test-table [id=63f2e3eedc5b418cba2f1fa38b496bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:30.434725 8210 tablet_service.cc:1467] Processing CreateTablet for tablet 5612384b173a48fd957e161393a9db5a (DEFAULT_TABLE table=test-table [id=63f2e3eedc5b418cba2f1fa38b496bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:30.435292 8489 tablet_service.cc:1467] Processing CreateTablet for tablet 5612384b173a48fd957e161393a9db5a (DEFAULT_TABLE table=test-table [id=63f2e3eedc5b418cba2f1fa38b496bc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:30.435853 8351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5612384b173a48fd957e161393a9db5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:30.436182 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5612384b173a48fd957e161393a9db5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:30.436648 8489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5612384b173a48fd957e161393a9db5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:30.447120 8573 tablet_bootstrap.cc:492] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Bootstrap starting.
I20260806 20:41:30.450505 8573 tablet_bootstrap.cc:654] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:30.450492 8574 tablet_bootstrap.cc:492] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Bootstrap starting.
I20260806 20:41:30.451951 8575 tablet_bootstrap.cc:492] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Bootstrap starting.
I20260806 20:41:30.452486 8573 log.cc:824] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:30.454243 8574 tablet_bootstrap.cc:654] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:30.455107 8575 tablet_bootstrap.cc:654] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:30.455405 8574 log.cc:824] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:30.456287 8575 log.cc:824] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:30.457123 8573 tablet_bootstrap.cc:492] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: No bootstrap required, opened a new log
I20260806 20:41:30.457389 8573 ts_tablet_manager.cc:1397] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260806 20:41:30.458072 8574 tablet_bootstrap.cc:492] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: No bootstrap required, opened a new log
I20260806 20:41:30.458437 8574 ts_tablet_manager.cc:1397] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260806 20:41:30.459396 8575 tablet_bootstrap.cc:492] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: No bootstrap required, opened a new log
I20260806 20:41:30.459635 8575 ts_tablet_manager.cc:1397] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 20:41:30.465834 8574 raft_consensus.cc:348] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.466189 8574 raft_consensus.cc:374] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:30.466342 8574 raft_consensus.cc:729] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d2365e9b614ee6b95cf8716741a4a5, State: Initialized, Role: FOLLOWER
I20260806 20:41:30.466933 8574 consensus_queue.cc:260] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.468041 8573 raft_consensus.cc:348] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.468534 8573 raft_consensus.cc:374] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:30.468667 8574 ts_tablet_manager.cc:1428] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Time spent starting tablet: real 0.010s user 0.002s sys 0.007s
I20260806 20:41:30.468668 8573 raft_consensus.cc:729] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315c5a139a3c4b718a9826c98f83cbdf, State: Initialized, Role: FOLLOWER
I20260806 20:41:30.469328 8573 consensus_queue.cc:260] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.470439 8418 heartbeater.cc:503] Master 127.23.200.62:45869 was elected leader, sending a full tablet report...
I20260806 20:41:30.471046 8573 ts_tablet_manager.cc:1428] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 20:41:30.470165 8575 raft_consensus.cc:348] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.471709 8575 raft_consensus.cc:374] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:30.471832 8575 raft_consensus.cc:729] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3a9ce1a6fd48a6a970666dec268aab, State: Initialized, Role: FOLLOWER
I20260806 20:41:30.472411 8575 consensus_queue.cc:260] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.473497 8555 heartbeater.cc:503] Master 127.23.200.62:45869 was elected leader, sending a full tablet report...
I20260806 20:41:30.474256 8575 ts_tablet_manager.cc:1428] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260806 20:41:30.575507 8579 raft_consensus.cc:482] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:30.575835 8579 raft_consensus.cc:504] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.577773 8579 leader_election.cc:288] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
I20260806 20:41:30.584558 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.578099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49815 (local address 127.23.200.2:33059)
0806 20:41:30.578439 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.578450 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.579213 (+ 763us) server_negotiation.cc:407] Connection header received
0806 20:41:30.579471 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.579476 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.579537 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.579633 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:30.580203 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.580937 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.582800 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.583629 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.583791 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.583871 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.584172 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.584295 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 20:41:30.584672 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.578337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:52831 (local address 127.23.200.3:37317)
0806 20:41:30.578565 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.578573 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.579404 (+ 831us) server_negotiation.cc:407] Connection header received
0806 20:41:30.579615 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.579622 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.579685 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.579781 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:30.580418 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.581138 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.582965 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.583729 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.583909 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.583992 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.584295 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.584418 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 20:41:30.584728 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.577943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33059 (local address 127.23.200.1:49815)
0806 20:41:30.579095 (+ 1152us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:30.579111 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:30.579294 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:30.579654 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:30.579659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:30.579685 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:30.580046 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.580059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.581085 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.581091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.582638 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.582650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.583840 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.583846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.584019 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.584044 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 20:41:30.584147 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 20:41:30.584271 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1041,"mutex_wait_us":2,"thread_start_us":796,"threads_started":1}
I20260806 20:41:30.584807 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.578282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37317 (local address 127.23.200.1:52831)
0806 20:41:30.579300 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:30.579313 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:30.579482 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:30.579800 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:30.579804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:30.579842 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:30.580266 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.580275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.581297 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.581304 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.582807 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:30.582820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.583951 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.583957 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:30.584141 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.584164 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:41:30.584267 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 20:41:30.584384 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":905,"mutex_wait_us":29,"thread_start_us":681,"threads_started":1}
I20260806 20:41:30.585825 8510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5612384b173a48fd957e161393a9db5a" candidate_uuid: "92d2365e9b614ee6b95cf8716741a4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" is_pre_election: true
I20260806 20:41:30.585850 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5612384b173a48fd957e161393a9db5a" candidate_uuid: "92d2365e9b614ee6b95cf8716741a4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315c5a139a3c4b718a9826c98f83cbdf" is_pre_election: true
I20260806 20:41:30.586349 8510 raft_consensus.cc:2454] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d2365e9b614ee6b95cf8716741a4a5 in term 0.
I20260806 20:41:30.586367 8371 raft_consensus.cc:2454] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92d2365e9b614ee6b95cf8716741a4a5 in term 0.
I20260806 20:41:30.587083 8163 leader_election.cc:302] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1d3a9ce1a6fd48a6a970666dec268aab, 315c5a139a3c4b718a9826c98f83cbdf, 92d2365e9b614ee6b95cf8716741a4a5; no voters:
I20260806 20:41:30.587519 8579 raft_consensus.cc:2781] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:41:30.587663 8579 raft_consensus.cc:482] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:41:30.587786 8579 raft_consensus.cc:3037] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:30.589977 8579 raft_consensus.cc:504] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:30.590713 8579 leader_election.cc:288] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 1 election: Requested vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
I20260806 20:41:30.591096 8371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5612384b173a48fd957e161393a9db5a" candidate_uuid: "92d2365e9b614ee6b95cf8716741a4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "315c5a139a3c4b718a9826c98f83cbdf"
I20260806 20:41:30.591194 8510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5612384b173a48fd957e161393a9db5a" candidate_uuid: "92d2365e9b614ee6b95cf8716741a4a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
I20260806 20:41:30.591298 8371 raft_consensus.cc:3037] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:30.591429 8510 raft_consensus.cc:3037] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:30.593931 8510 raft_consensus.cc:2454] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d2365e9b614ee6b95cf8716741a4a5 in term 1.
I20260806 20:41:30.594146 8371 raft_consensus.cc:2454] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92d2365e9b614ee6b95cf8716741a4a5 in term 1.
I20260806 20:41:30.594348 8163 leader_election.cc:302] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d3a9ce1a6fd48a6a970666dec268aab, 92d2365e9b614ee6b95cf8716741a4a5; no voters:
I20260806 20:41:30.594627 8579 raft_consensus.cc:2781] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:30.595027 8579 raft_consensus.cc:686] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 LEADER]: Becoming Leader. State: Replica: 92d2365e9b614ee6b95cf8716741a4a5, State: Running, Role: LEADER
I20260806 20:41:30.595485 8579 consensus_queue.cc:237] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
W20260806 20:41:30.595793 8419 tablet.cc:2369] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:41:30.596170 8556 tablet.cc:2369] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:41:30.599130 8278 tablet.cc:2369] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:41:30.600808 8083 catalog_manager.cc:5495] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1). New cstate: current_term: 1 leader_uuid: "92d2365e9b614ee6b95cf8716741a4a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
I20260806 20:41:30.621889 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.616373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59316 (local address 127.23.200.1:39649)
0806 20:41:30.616588 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.616594 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.616702 (+ 108us) server_negotiation.cc:407] Connection header received
0806 20:41:30.616938 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.616950 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.617027 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.617176 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:41:30.617568 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.618280 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.619764 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.620217 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.620353 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.620844 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:41:30.621322 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:41:30.621485 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.621551 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.621630 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 20:41:30.629590 8510 raft_consensus.cc:1261] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Refusing update from remote peer 92d2365e9b614ee6b95cf8716741a4a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:30.629596 8371 raft_consensus.cc:1261] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Refusing update from remote peer 92d2365e9b614ee6b95cf8716741a4a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:30.630595 8584 consensus_queue.cc:1035] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:30.630919 8579 consensus_queue.cc:1035] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:30.641733 8587 mvcc.cc:205] Tried to move back new op lower bound from 7315656256009175040 to 7315656255883493376. Current Snapshot: MvccSnapshot[applied={T|T < 7315656256009175040}]
I20260806 20:41:30.657544 8084 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:30.657814 8084 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:30.661504 8084 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Sending DeleteTablet for 3 replicas of tablet 5612384b173a48fd957e161393a9db5a
I20260806 20:41:30.662612 8351 tablet_service.cc:1514] Processing DeleteTablet for tablet 5612384b173a48fd957e161393a9db5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:30 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:44422
I20260806 20:41:30.662684 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet 5612384b173a48fd957e161393a9db5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:30 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:59310
I20260806 20:41:30.663080 8489 tablet_service.cc:1514] Processing DeleteTablet for tablet 5612384b173a48fd957e161393a9db5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:30 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:38204
I20260806 20:41:30.664147 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8146
I20260806 20:41:30.690644 8601 tablet_replica.cc:330] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: stopping tablet replica
I20260806 20:41:30.692569 8601 raft_consensus.cc:2229] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:30.693795 8601 raft_consensus.cc:2258] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:30.698885 8601 ts_tablet_manager.cc:1905] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:30.701534 8599 tablet_replica.cc:330] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: stopping tablet replica
I20260806 20:41:30.703222 8599 raft_consensus.cc:2229] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:30.707015 8599 raft_consensus.cc:2258] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:30.710541 8599 ts_tablet_manager.cc:1905] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:30.715806 8601 ts_tablet_manager.cc:1918] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:30.716483 8600 tablet_replica.cc:330] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: stopping tablet replica
I20260806 20:41:30.716958 8600 raft_consensus.cc:2229] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:41:30.717705 8600 raft_consensus.cc:2258] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:30.718178 8599 ts_tablet_manager.cc:1918] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:30.718909 8599 log.cc:1196] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/5612384b173a48fd957e161393a9db5a
Aug 06 20:41:30 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:41:30.719607 8599 ts_tablet_manager.cc:1939] T 5612384b173a48fd957e161393a9db5a P 315c5a139a3c4b718a9826c98f83cbdf: Deleting consensus metadata
I20260806 20:41:30.720587 8600 ts_tablet_manager.cc:1905] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:30.721617 8070 catalog_manager.cc:4841] TS 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): tablet 5612384b173a48fd957e161393a9db5a (table test-table [id=63f2e3eedc5b418cba2f1fa38b496bc1]) successfully deleted
I20260806 20:41:30.726538 8601 log.cc:1196] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/5612384b173a48fd957e161393a9db5a
I20260806 20:41:30.727123 8601 ts_tablet_manager.cc:1939] T 5612384b173a48fd957e161393a9db5a P 1d3a9ce1a6fd48a6a970666dec268aab: Deleting consensus metadata
I20260806 20:41:30.728863 8600 ts_tablet_manager.cc:1918] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:30.729000 8069 catalog_manager.cc:4841] TS 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): tablet 5612384b173a48fd957e161393a9db5a (table test-table [id=63f2e3eedc5b418cba2f1fa38b496bc1]) successfully deleted
I20260806 20:41:30.729105 8600 log.cc:1196] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/5612384b173a48fd957e161393a9db5a
I20260806 20:41:30.729645 8600 ts_tablet_manager.cc:1939] T 5612384b173a48fd957e161393a9db5a P 92d2365e9b614ee6b95cf8716741a4a5: Deleting consensus metadata
I20260806 20:41:30.730129 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.665235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.23.200.1:39649)
0806 20:41:30.665420 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.665426 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.689927 (+ 24501us) server_negotiation.cc:407] Connection header received
0806 20:41:30.691138 (+ 1211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.691144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.691308 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.691392 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:30.695875 (+ 4483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.697144 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.703976 (+ 6832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.704723 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.704831 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.720860 (+ 16029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:30.720875 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:30.720893 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:30.720957 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:30.725631 (+ 4674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.726676 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.726685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.726689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.726875 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.727369 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.727375 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.727378 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.727864 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:30.728166 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.729470 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.729734 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:41:30.731736 8071 catalog_manager.cc:4841] TS 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1:39649): tablet 5612384b173a48fd957e161393a9db5a (table test-table [id=63f2e3eedc5b418cba2f1fa38b496bc1]) successfully deleted
I20260806 20:41:30.732141 8272 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:30.847625 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8146
I20260806 20:41:30.874173 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8284
Aug 06 20:41:30 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:41:30.890129 8569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:30.875086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44424 (local address 127.23.200.2:33059)
0806 20:41:30.875262 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:30.875271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:30.875293 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:41:30.875541 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:30.875549 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:30.875616 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:30.875723 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:30.877078 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.878007 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.879634 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:30.880035 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:30.880169 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:30.884474 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:30.884491 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:30.884509 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:30.884551 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:30.887282 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.887955 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.887971 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.887973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.888102 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:30.888519 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:30.888525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:30.888528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:30.888920 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:30.889227 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:30.889558 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 20:41:30.889780 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:41:30.890790 8413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:30.981787 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8284
I20260806 20:41:31.000622 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8423
Aug 06 20:41:31 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:41:31.016244 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:31.001489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38208 (local address 127.23.200.3:37317)
0806 20:41:31.001681 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:31.001685 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:31.001753 (+ 68us) server_negotiation.cc:407] Connection header received
0806 20:41:31.001963 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:31.001968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:31.002043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:31.002154 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:31.003667 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:31.004478 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:31.005833 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:31.006228 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:31.006389 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:31.010799 (+ 4410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:31.010815 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:31.010833 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:31.010873 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:31.013399 (+ 2526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:31.014117 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:31.014124 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:31.014127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:31.014270 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:31.014698 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:31.014702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:31.014708 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:31.015045 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:31.015349 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:31.015672 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 20:41:31.015897 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 20:41:31.016868 8550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:31.105444 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8423
I20260806 20:41:31.123458 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8052
I20260806 20:41:31.124652 8115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:31.234735 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8052
I20260806 20:41:51.257370 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45869
--webserver_interface=127.23.200.62
--webserver_port=44493
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:41:51.589591 8608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:51.589867 8608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:51.589956 8608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:51.597116 8608 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 20:41:51.597213 8608 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 20:41:51.597251 8608 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 20:41:51.597311 8608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:41:51.597363 8608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:51.597426 8608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:51.597470 8608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:41:51.597523 8608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:41:51.607266 8608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=44493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:51.608528 8608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:51.609812 8608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:51.632892 8617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:51.632869 8615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:51.632965 8614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:51.633772 8608 server_base.cc:1034] running on GCE node
I20260806 20:41:51.634809 8608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:51.636422 8608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:51.637706 8608 hybrid_clock.cc:648] HybridClock initialized: now 1786048911637646 us; error 44 us; skew 500 ppm
I20260806 20:41:51.638152 8608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:51 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048911, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:51.642045 8608 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:41:51.643887 8608 webserver.cc:460] Webserver started at http://127.23.200.62:44493/ using document root <none> and password file <none>
I20260806 20:41:51.644794 8608 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:51.645028 8608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:51.651382 8608 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 20:41:51.654529 8625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:51.655885 8608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:41:51.656081 8608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "fd1d4d6e9f5249928d329c78a7464f02"
format_stamp: "Formatted at 2026-08-06 20:41:28 on dist-test-slave-tm1g"
server_key: "1434dc336f9d4821256cea2bf90e3bc7"
server_key_iv: "f47ab5fa85581688763ab1eef6612b84"
server_key_version: "encryptionkey@0"
I20260806 20:41:51.657227 8608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:51.679639 8608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:51.683995 8608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:51.684370 8608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:51.705096 8608 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:45869
I20260806 20:41:51.705134 8677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:45869 every 8 connection(s)
I20260806 20:41:51.706784 8608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:41:51.711803 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8608
I20260806 20:41:51.712656 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:39649
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:45869
--webserver_port=41157
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:41:51.713840 8678 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:41:51.717165 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap starting.
I20260806 20:41:51.743889 8678 log.cc:824] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:51.757627 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:41:51.758183 8678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap complete.
I20260806 20:41:51.767786 8678 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:51.768644 8678 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d4d6e9f5249928d329c78a7464f02, State: Initialized, Role: FOLLOWER
I20260806 20:41:51.769222 8678 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:51.769479 8678 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:41:51.769609 8678 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:41:51.769780 8678 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 1 FOLLOWER]: Advancing to term 2
I20260806 20:41:51.773564 8678 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:51.774073 8678 leader_election.cc:302] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1d4d6e9f5249928d329c78a7464f02; no voters:
I20260806 20:41:51.774771 8678 leader_election.cc:288] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 20:41:51.774951 8682 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 20:41:51.776196 8682 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 LEADER]: Becoming Leader. State: Replica: fd1d4d6e9f5249928d329c78a7464f02, State: Running, Role: LEADER
I20260806 20:41:51.777007 8678 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:41:51.776849 8682 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:41:51.780076 8684 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1d4d6e9f5249928d329c78a7464f02. Latest consensus state: current_term: 2 leader_uuid: "fd1d4d6e9f5249928d329c78a7464f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } } }
I20260806 20:41:51.780435 8684 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:51.781260 8683 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd1d4d6e9f5249928d329c78a7464f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } } }
I20260806 20:41:51.781628 8683 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: This master's current role is: LEADER
I20260806 20:41:51.787361 8689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:41:51.795598 8689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:41:51.796917 8689 catalog_manager.cc:1259] Loaded cluster ID: 8cf08da2847a45959ac98fc46c8102fc
I20260806 20:41:51.797034 8689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:41:51.799731 8689 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:41:51.800814 8689 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Loaded TSK: 0
I20260806 20:41:51.824174 8689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Generated new TSK 1
I20260806 20:41:51.824524 8689 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 20:41:52.083343 8680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:52.083612 8680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:52.083765 8680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:52.091197 8680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:52.091298 8680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:52.091470 8680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:41:52.101076 8680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:39649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=41157
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:52.102353 8680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:52.103689 8680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:52.118749 8709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:52.118988 8706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:52.118793 8707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:52.120219 8680 server_base.cc:1034] running on GCE node
I20260806 20:41:52.120887 8680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:52.121892 8680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:52.123112 8680 hybrid_clock.cc:648] HybridClock initialized: now 1786048912123068 us; error 41 us; skew 500 ppm
I20260806 20:41:52.123435 8680 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:52 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:52.126701 8680 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:41:52.128170 8680 webserver.cc:460] Webserver started at http://127.23.200.1:41157/ using document root <none> and password file <none>
I20260806 20:41:52.128859 8680 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:52.129020 8680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:52.134562 8680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:52.137245 8717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:52.138713 8680 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:52.138888 8680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "92d2365e9b614ee6b95cf8716741a4a5"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "5f180a46d5c2e2a79cbe547017116462"
server_key_iv: "60f37531562cb345afe32f27027941e8"
server_key_version: "encryptionkey@0"
I20260806 20:41:52.139846 8680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:52.153765 8680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:52.157868 8680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:52.158154 8680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:52.159158 8680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:52.160561 8680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:52.160650 8680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:52.160796 8680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:52.160889 8680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:52.194832 8680 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:39649
I20260806 20:41:52.194923 8835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:39649 every 8 connection(s)
I20260806 20:41:52.196324 8680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:41:52.202454 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8680
I20260806 20:41:52.203227 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:33059
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:45869
--webserver_port=41389
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:41:52 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:52.220510 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:52.199138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:42257 (local address 127.23.200.62:45869)
0806 20:41:52.199774 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:52.199785 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:52.200304 (+ 519us) server_negotiation.cc:407] Connection header received
0806 20:41:52.202557 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:52.202580 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:52.203016 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:52.203436 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:52.204872 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.206898 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.208418 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.209491 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.209691 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:52.213067 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:52.213075 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:52.213088 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:52.213124 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:52.216153 (+ 3029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:52.217081 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:52.217087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:52.217092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:52.217251 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:52.217738 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:52.217741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:52.217743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:52.218174 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:52.218383 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:52.219263 (+ 880us) server_negotiation.cc:299] Negotiation successful
0806 20:41:52.219579 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":278,"threads_started":1}
I20260806 20:41:52.220983 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:52.199290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:42257)
0806 20:41:52.200120 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:52.200172 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:52.201907 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:52.203631 (+ 1724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:52.203638 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:52.204140 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:52.204669 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:52.204685 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.207061 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.207065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:52.208269 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:52.208281 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.209709 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.209713 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:52.209897 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:52.210600 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:52.210622 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:52.212786 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:52.216298 (+ 3512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:52.216310 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:52.216326 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:52.216946 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:52.217377 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:52.217384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:52.217391 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:52.217627 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:52.218346 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:52.218357 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:52.218794 (+ 437us) client_negotiation.cc:769] Sending connection context
0806 20:41:52.219106 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 20:41:52.219705 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":358,"threads_started":1}
I20260806 20:41:52.223912 8836 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45869
I20260806 20:41:52.224267 8836 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:52.225232 8836 heartbeater.cc:511] Master 127.23.200.62:45869 requested a full tablet report, sending...
I20260806 20:41:52.227635 8642 ts_manager.cc:194] Registered new tserver with Master: 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1:39649)
I20260806 20:41:52.229161 8642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:42257
W20260806 20:41:52.538960 8840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:52.539304 8840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:52.539487 8840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:52.546640 8840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:52.546774 8840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:52.546969 8840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:41:52.556411 8840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:33059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=41389
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:52.557641 8840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:52.558943 8840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:52.571911 8845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:52.571842 8846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:52.572017 8848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:52.572844 8840 server_base.cc:1034] running on GCE node
I20260806 20:41:52.573464 8840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:52.574358 8840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:52.575589 8840 hybrid_clock.cc:648] HybridClock initialized: now 1786048912575531 us; error 56 us; skew 500 ppm
I20260806 20:41:52.575906 8840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:52 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:52.579083 8840 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:41:52.580459 8840 webserver.cc:460] Webserver started at http://127.23.200.2:41389/ using document root <none> and password file <none>
I20260806 20:41:52.581127 8840 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:52.581238 8840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:52.586683 8840 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260806 20:41:52.589372 8855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:52.590826 8840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:41:52.591007 8840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "315c5a139a3c4b718a9826c98f83cbdf"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "b4de5d11143a868355ce652d3b4fecdb"
server_key_iv: "fae544854574a8d446d08bfca2ec0ca0"
server_key_version: "encryptionkey@0"
I20260806 20:41:52.591996 8840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:52.611944 8840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:52.616250 8840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:52.616560 8840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:52.617570 8840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:52.618996 8840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:52.619122 8840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:52.619264 8840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:52.619365 8840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:52.653209 8840 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33059
I20260806 20:41:52.653297 8973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33059 every 8 connection(s)
I20260806 20:41:52.654835 8840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:41:52.661372 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8840
I20260806 20:41:52.662086 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:37317
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:45869
--webserver_port=36605
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 20:41:52 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048912, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:52.674788 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:52.657386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:47733 (local address 127.23.200.62:45869)
0806 20:41:52.657583 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:52.657588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:52.658832 (+ 1244us) server_negotiation.cc:407] Connection header received
0806 20:41:52.660644 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:52.660650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:52.660740 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:52.660851 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:52.662576 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.663293 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.664861 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.665283 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.665438 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:52.669154 (+ 3716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:52.669159 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:52.669162 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:52.669196 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:52.671777 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:52.672512 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:52.672523 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:52.672526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:52.672645 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:52.673082 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:52.673087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:52.673090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:52.673367 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:52.673520 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:52.674186 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 20:41:52.674451 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:41:52.675719 8976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:52.657882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:47733)
0806 20:41:52.658673 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:52.658711 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:52.660380 (+ 1669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:52.661088 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:52.661097 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:52.661740 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:52.662382 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:52.662427 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.663474 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.663480 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:52.664704 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:52.664715 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:52.665547 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:52.665554 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:52.665791 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:52.666508 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:52.666535 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:52.668919 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:52.671935 (+ 3016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:52.671943 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:52.671957 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:52.672379 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:52.672764 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:52.672769 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:52.672773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:52.672961 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:52.673503 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:52.673511 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:52.673832 (+ 321us) client_negotiation.cc:769] Sending connection context
0806 20:41:52.674090 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 20:41:52.674609 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":292,"threads_started":1}
I20260806 20:41:52.677204 8974 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45869
I20260806 20:41:52.677542 8974 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:52.678288 8974 heartbeater.cc:511] Master 127.23.200.62:45869 requested a full tablet report, sending...
I20260806 20:41:52.679797 8642 ts_manager.cc:194] Registered new tserver with Master: 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059)
I20260806 20:41:52.681105 8642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:47733
W20260806 20:41:52.997589 8977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:41:52.997843 8977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:41:52.997934 8977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:41:53.006196 8977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:41:53.006294 8977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:41:53.006482 8977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:41:53.015965 8977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:37317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=36605
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:41:53.017164 8977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:41:53.018496 8977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:41:53.031723 8982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:53.031682 8983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:41:53.031764 8985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:41:53.032598 8977 server_base.cc:1034] running on GCE node
I20260806 20:41:53.033121 8977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:41:53.033960 8977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:41:53.035151 8977 hybrid_clock.cc:648] HybridClock initialized: now 1786048913035093 us; error 47 us; skew 500 ppm
I20260806 20:41:53.035429 8977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:41:53 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:41:53.038573 8977 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:41:53.039872 8977 webserver.cc:460] Webserver started at http://127.23.200.3:36605/ using document root <none> and password file <none>
I20260806 20:41:53.040498 8977 fs_manager.cc:359] Metadata directory not provided
I20260806 20:41:53.040606 8977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:41:53.045779 8977 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:41:53.048362 8992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:53.049635 8977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:41:53.049760 8977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
format_stamp: "Formatted at 2026-08-06 20:41:30 on dist-test-slave-tm1g"
server_key: "95d9305eab4ee0ae468cdf6ee625a839"
server_key_iv: "864adff332f253f1a47b5cd7d98f6178"
server_key_version: "encryptionkey@0"
I20260806 20:41:53.050735 8977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:41:53.063633 8977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:41:53.067677 8977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:41:53.067911 8977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:41:53.068742 8977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:41:53.069937 8977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:41:53.070030 8977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:53.070098 8977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:41:53.070154 8977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:41:53.104501 8977 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37317
I20260806 20:41:53.104584 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37317 every 8 connection(s)
I20260806 20:41:53.105957 8977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:41:53.107712 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 8977
Aug 06 20:41:53 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048913, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:41:53.128432 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.108772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44728 (local address 127.23.200.62:45869)
0806 20:41:53.114695 (+ 5923us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.114700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.114716 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:41:53.114756 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.114760 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.114814 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.114941 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:53.116516 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.117295 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.118817 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.119283 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.119416 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.122263 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:53.122268 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:53.122271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:53.122355 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:53.125596 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.126280 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.126286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.126290 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.126456 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.126835 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.126841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.126845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.127142 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:53.127396 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.127731 (+ 335us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.127978 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":124,"server-negotiator.queue_time_us":5794,"thread_start_us":5557,"threads_started":1}
I20260806 20:41:53.128649 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.108772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:47101 (local address 127.23.200.62:45869)
0806 20:41:53.109111 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.109120 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.110911 (+ 1791us) server_negotiation.cc:407] Connection header received
0806 20:41:53.114464 (+ 3553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.114473 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.114575 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.114685 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:53.116125 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.116903 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.118484 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.118971 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.119109 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.122387 (+ 3278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:53.122399 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:53.122402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:53.122445 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:53.125645 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.126364 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.126369 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.126371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.126483 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.126924 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.126929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.126932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.127208 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:53.127364 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.128004 (+ 640us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.128209 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":175}
I20260806 20:41:53.129755 9117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.109552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:47101)
0806 20:41:53.110697 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.110765 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:53.114081 (+ 3316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:53.114880 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:53.114888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:53.115404 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:41:53.115955 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.115970 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.117063 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.117070 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.118288 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.118348 (+ 60us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.119191 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.119201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.119458 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.120110 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:41:53.120131 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:41:53.122165 (+ 2034us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:41:53.125783 (+ 3618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:53.125791 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:53.125805 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:53.126209 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:53.126604 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:41:53.126609 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:41:53.126612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:41:53.126793 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:41:53.127338 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:41:53.127346 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:41:53.127667 (+ 321us) client_negotiation.cc:769] Sending connection context
0806 20:41:53.127926 (+ 259us) client_negotiation.cc:240] Negotiation successful
0806 20:41:53.128439 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":405,"threads_started":1}
I20260806 20:41:53.131220 9113 heartbeater.cc:347] Connected to a master server at 127.23.200.62:45869
I20260806 20:41:53.131538 9113 heartbeater.cc:464] Registering TS with master...
I20260806 20:41:53.132241 9113 heartbeater.cc:511] Master 127.23.200.62:45869 requested a full tablet report, sending...
I20260806 20:41:53.133589 8642 ts_manager.cc:194] Registered new tserver with Master: 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
I20260806 20:41:53.134454 8642 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:47101
I20260806 20:41:53.143467 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:41:53.149819 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.144380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.23.200.62:45869)
0806 20:41:53.144584 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.144588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.145073 (+ 485us) server_negotiation.cc:407] Connection header received
0806 20:41:53.145239 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.145242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.145384 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.145512 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:41:53.145906 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.146645 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.148134 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.148548 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.148641 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.149112 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:41:53.149349 (+ 237us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 20:41:53.149632 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44742: token expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260806 20:41:53.149955 9116 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44742: token expired
W20260806 20:41:53.150565 8560 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.200.62:45869)
I20260806 20:41:53.163960 9116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.151197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44752 (local address 127.23.200.62:45869)
0806 20:41:53.151421 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.151427 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.151441 (+ 14us) server_negotiation.cc:407] Connection header received
0806 20:41:53.151477 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.151480 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.151530 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.151647 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:53.153032 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.153838 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.155334 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.155732 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.155874 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.158583 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:53.158588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:53.158590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:53.158619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:53.161079 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.161768 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.161776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.161779 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.161918 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.162335 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.162347 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.162349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.162823 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:53.163096 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.163423 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.163638 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 20:41:53.167598 8642 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:41:53.175050 8642 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:41:53.208668 9125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.186982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36010)
0806 20:41:53.199660 (+ 12678us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.199709 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:53.199899 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:53.201712 (+ 1813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:53.201717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:53.201742 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:53.202161 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.202174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.204122 (+ 1948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.204128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.205992 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.206013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.207717 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.207723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.207910 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.207959 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 20:41:53.208099 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 20:41:53.208274 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12395,"thread_start_us":13074,"threads_started":1}
I20260806 20:41:53.209770 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.187041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36010 (local address 127.23.200.1:39649)
0806 20:41:53.199449 (+ 12408us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.199459 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.199930 (+ 471us) server_negotiation.cc:407] Connection header received
0806 20:41:53.201021 (+ 1091us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.201029 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.201237 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.201423 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:53.202374 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.203949 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.206170 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.207492 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.207635 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.207880 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.209252 (+ 1372us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.209427 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12231,"thread_start_us":12119,"threads_started":1}
I20260806 20:41:53.209817 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.189395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37317 (local address 127.0.0.1:35038)
0806 20:41:53.200651 (+ 11256us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.200669 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:53.200856 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:53.202162 (+ 1306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:53.202167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:53.202193 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:53.202569 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.202580 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.205401 (+ 2821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.205409 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.207228 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.207239 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.209017 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.209024 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.209207 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.209239 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 20:41:53.209353 (+ 114us) client_negotiation.cc:240] Negotiation successful
0806 20:41:53.209489 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11129,"thread_start_us":10976,"threads_started":1}
I20260806 20:41:53.209900 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.190173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.23.200.3:37317)
0806 20:41:53.195965 (+ 5792us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.195974 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.200759 (+ 4785us) server_negotiation.cc:407] Connection header received
0806 20:41:53.201050 (+ 291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.201058 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.201265 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.201447 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:53.202768 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.205169 (+ 2401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.207366 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.208745 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.208878 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.209117 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.209416 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.209562 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5593,"thread_start_us":8134,"threads_started":1}
I20260806 20:41:53.210397 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.187434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.23.200.2:33059)
0806 20:41:53.198772 (+ 11338us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.198783 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.199850 (+ 1067us) server_negotiation.cc:407] Connection header received
0806 20:41:53.200224 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.200236 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.200511 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.200708 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:53.202067 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.203772 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.206785 (+ 3013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.208216 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.208377 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.208572 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.209632 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.209807 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11112,"thread_start_us":11329,"threads_started":1}
I20260806 20:41:53.211019 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.187060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33059 (local address 127.0.0.1:48284)
0806 20:41:53.199711 (+ 12651us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.199726 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:53.199903 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:53.201468 (+ 1565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:53.201477 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:53.201512 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:53.201882 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.201896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.203956 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.203963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.206621 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.206636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.208746 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.208753 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.209411 (+ 658us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.209445 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 20:41:53.210383 (+ 938us) client_negotiation.cc:240] Negotiation successful
0806 20:41:53.210625 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12547,"spinlock_wait_cycles":4480,"thread_start_us":12167,"threads_started":1}
I20260806 20:41:53.213423 8767 tablet_service.cc:1467] Processing CreateTablet for tablet 6559a47b8c1544749a25c66fe72e8d6b (DEFAULT_TABLE table=test-table [id=10acad3baea14387a37e34bdc4bfb5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:53.213536 9044 tablet_service.cc:1467] Processing CreateTablet for tablet 6559a47b8c1544749a25c66fe72e8d6b (DEFAULT_TABLE table=test-table [id=10acad3baea14387a37e34bdc4bfb5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:53.214344 8906 tablet_service.cc:1467] Processing CreateTablet for tablet 6559a47b8c1544749a25c66fe72e8d6b (DEFAULT_TABLE table=test-table [id=10acad3baea14387a37e34bdc4bfb5e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:41:53.214929 8767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6559a47b8c1544749a25c66fe72e8d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:53.214946 9044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6559a47b8c1544749a25c66fe72e8d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:53.215586 8906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6559a47b8c1544749a25c66fe72e8d6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:41:53.228360 9129 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Bootstrap starting.
I20260806 20:41:53.230235 9130 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Bootstrap starting.
I20260806 20:41:53.231895 9131 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Bootstrap starting.
I20260806 20:41:53.232372 9129 tablet_bootstrap.cc:654] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:53.232771 8836 heartbeater.cc:503] Master 127.23.200.62:45869 was elected leader, sending a full tablet report...
I20260806 20:41:53.233755 9129 log.cc:824] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:53.234833 9131 tablet_bootstrap.cc:654] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:53.235494 9130 tablet_bootstrap.cc:654] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Neither blocks nor log segments found. Creating new log.
I20260806 20:41:53.236064 9131 log.cc:824] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:53.237000 9130 log.cc:824] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Log is configured to *not* fsync() on all Append() calls
I20260806 20:41:53.240924 9129 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: No bootstrap required, opened a new log
I20260806 20:41:53.241190 9129 ts_tablet_manager.cc:1397] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260806 20:41:53.241286 9131 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: No bootstrap required, opened a new log
I20260806 20:41:53.241533 9131 ts_tablet_manager.cc:1397] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260806 20:41:53.241782 9130 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: No bootstrap required, opened a new log
I20260806 20:41:53.242039 9130 ts_tablet_manager.cc:1397] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260806 20:41:53.249594 9130 raft_consensus.cc:348] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.249969 9130 raft_consensus.cc:374] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:53.250108 9130 raft_consensus.cc:729] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 315c5a139a3c4b718a9826c98f83cbdf, State: Initialized, Role: FOLLOWER
I20260806 20:41:53.250765 9130 consensus_queue.cc:260] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.251855 8974 heartbeater.cc:503] Master 127.23.200.62:45869 was elected leader, sending a full tablet report...
I20260806 20:41:53.252079 9131 raft_consensus.cc:348] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.252079 9129 raft_consensus.cc:348] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.252497 9131 raft_consensus.cc:374] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:53.252497 9129 raft_consensus.cc:374] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:41:53.252583 9130 ts_tablet_manager.cc:1428] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20260806 20:41:53.252589 9131 raft_consensus.cc:729] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d3a9ce1a6fd48a6a970666dec268aab, State: Initialized, Role: FOLLOWER
I20260806 20:41:53.252589 9129 raft_consensus.cc:729] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d2365e9b614ee6b95cf8716741a4a5, State: Initialized, Role: FOLLOWER
I20260806 20:41:53.253158 9131 consensus_queue.cc:260] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.253158 9129 consensus_queue.cc:260] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.254245 9113 heartbeater.cc:503] Master 127.23.200.62:45869 was elected leader, sending a full tablet report...
I20260806 20:41:53.255045 9131 ts_tablet_manager.cc:1428] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260806 20:41:53.256693 9129 ts_tablet_manager.cc:1428] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Time spent starting tablet: real 0.015s user 0.016s sys 0.001s
I20260806 20:41:53.321328 9135 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:41:53.321653 9135 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.323482 9135 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1:39649), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
I20260806 20:41:53.330422 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.324626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:48005 (local address 127.23.200.3:37317)
0806 20:41:53.324842 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.324848 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.324920 (+ 72us) server_negotiation.cc:407] Connection header received
0806 20:41:53.325079 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.325084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.325155 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.325248 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:53.325812 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.326678 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.328602 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.329421 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.329597 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.329704 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.329993 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.330125 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:41:53.330448 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.323961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:43799 (local address 127.23.200.1:39649)
0806 20:41:53.324186 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.324195 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.324551 (+ 356us) server_negotiation.cc:407] Connection header received
0806 20:41:53.324886 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.324894 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.324986 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.325069 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:41:53.326012 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.326900 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.328671 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.329423 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.329605 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.329690 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.330058 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.330189 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 20:41:53.330569 9139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.323984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37317 (local address 127.23.200.2:48005)
0806 20:41:53.324831 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.324845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:53.324955 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:53.325266 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:53.325269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:53.325288 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:53.325648 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.325660 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.326812 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.326818 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.328442 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.328457 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.329662 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.329669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.329834 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.329858 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 20:41:53.329935 (+ 77us) client_negotiation.cc:240] Negotiation successful
0806 20:41:53.330064 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"mutex_wait_us":56,"thread_start_us":522,"threads_started":1}
I20260806 20:41:53.330605 9138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.323728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.23.200.2:43799)
0806 20:41:53.324424 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.324440 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:41:53.324690 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:41:53.325277 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:41:53.325281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:41:53.325322 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:41:53.325825 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.325847 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.327037 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.327041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.328495 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:41:53.328531 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.329728 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.329735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:41:53.329880 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.329902 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:41:53.330015 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 20:41:53.330134 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":2,"thread_start_us":366,"threads_started":1}
I20260806 20:41:53.332001 9066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6559a47b8c1544749a25c66fe72e8d6b" candidate_uuid: "315c5a139a3c4b718a9826c98f83cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" is_pre_election: true
I20260806 20:41:53.332001 8787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6559a47b8c1544749a25c66fe72e8d6b" candidate_uuid: "315c5a139a3c4b718a9826c98f83cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d2365e9b614ee6b95cf8716741a4a5" is_pre_election: true
I20260806 20:41:53.332571 8787 raft_consensus.cc:2454] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315c5a139a3c4b718a9826c98f83cbdf in term 0.
I20260806 20:41:53.332571 9066 raft_consensus.cc:2454] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 315c5a139a3c4b718a9826c98f83cbdf in term 0.
I20260806 20:41:53.333397 8859 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315c5a139a3c4b718a9826c98f83cbdf, 92d2365e9b614ee6b95cf8716741a4a5; no voters:
I20260806 20:41:53.333827 9135 raft_consensus.cc:2781] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:41:53.333986 9135 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:41:53.334101 9135 raft_consensus.cc:3037] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:53.336311 9135 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.336954 9135 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [CANDIDATE]: Term 1 election: Requested vote from peers 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1:39649), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
I20260806 20:41:53.337337 8787 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6559a47b8c1544749a25c66fe72e8d6b" candidate_uuid: "315c5a139a3c4b718a9826c98f83cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92d2365e9b614ee6b95cf8716741a4a5"
I20260806 20:41:53.337435 9066 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6559a47b8c1544749a25c66fe72e8d6b" candidate_uuid: "315c5a139a3c4b718a9826c98f83cbdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
I20260806 20:41:53.337545 8787 raft_consensus.cc:3037] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:53.337678 9066 raft_consensus.cc:3037] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:41:53.339848 8787 raft_consensus.cc:2454] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315c5a139a3c4b718a9826c98f83cbdf in term 1.
I20260806 20:41:53.339956 9066 raft_consensus.cc:2454] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 315c5a139a3c4b718a9826c98f83cbdf in term 1.
I20260806 20:41:53.340233 8859 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 315c5a139a3c4b718a9826c98f83cbdf, 92d2365e9b614ee6b95cf8716741a4a5; no voters:
I20260806 20:41:53.340517 9135 raft_consensus.cc:2781] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:41:53.340952 9135 raft_consensus.cc:686] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 1 LEADER]: Becoming Leader. State: Replica: 315c5a139a3c4b718a9826c98f83cbdf, State: Running, Role: LEADER
I20260806 20:41:53.341410 9135 consensus_queue.cc:237] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:41:53.346278 8641 catalog_manager.cc:5495] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2). New cstate: current_term: 1 leader_uuid: "315c5a139a3c4b718a9826c98f83cbdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } health_report { overall_health: UNKNOWN } } }
W20260806 20:41:53.358578 9114 tablet.cc:2369] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:41:53.362895 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.357567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48296 (local address 127.23.200.2:33059)
0806 20:41:53.357784 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.357790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.357810 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:41:53.357956 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.357960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.358044 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.358187 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:41:53.358585 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.359348 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.360727 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.361139 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.361273 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.361800 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:41:53.362261 (+ 461us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:41:53.362495 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.362553 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.362629 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 20:41:53.370733 8787 raft_consensus.cc:1261] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Refusing update from remote peer 315c5a139a3c4b718a9826c98f83cbdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:53.370729 9066 raft_consensus.cc:1261] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Refusing update from remote peer 315c5a139a3c4b718a9826c98f83cbdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:41:53.371665 9140 consensus_queue.cc:1035] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:53.372020 9135 consensus_queue.cc:1035] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:41:53.383821 9144 mvcc.cc:205] Tried to move back new op lower bound from 7315656349156900864 to 7315656349050626048. Current Snapshot: MvccSnapshot[applied={T|T < 7315656349156900864}]
I20260806 20:41:53.401333 8641 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:53.401629 8641 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:41:53.404974 8641 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Sending DeleteTablet for 3 replicas of tablet 6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:41:53.406086 8767 tablet_service.cc:1514] Processing DeleteTablet for tablet 6559a47b8c1544749a25c66fe72e8d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:53 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:36010
I20260806 20:41:53.406096 8906 tablet_service.cc:1514] Processing DeleteTablet for tablet 6559a47b8c1544749a25c66fe72e8d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:53 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:48284
I20260806 20:41:53.406318 9044 tablet_service.cc:1514] Processing DeleteTablet for tablet 6559a47b8c1544749a25c66fe72e8d6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:41:53 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35038
I20260806 20:41:53.406626 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8680
I20260806 20:41:53.414803 9155 tablet_replica.cc:330] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: stopping tablet replica
I20260806 20:41:53.415307 9155 raft_consensus.cc:2229] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:41:53.418749 9155 raft_consensus.cc:2258] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:53.419919 9156 tablet_replica.cc:330] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: stopping tablet replica
I20260806 20:41:53.420377 9156 raft_consensus.cc:2229] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:53.420920 9156 raft_consensus.cc:2258] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:53.421391 9155 ts_tablet_manager.cc:1905] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:53.423228 9156 ts_tablet_manager.cc:1905] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:41:53.423969 9154 tablet_replica.cc:330] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: stopping tablet replica
I20260806 20:41:53.425575 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.407497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.23.200.1:39649)
0806 20:41:53.407675 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.407683 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.407803 (+ 120us) server_negotiation.cc:407] Connection header received
0806 20:41:53.407991 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.407999 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.408126 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.408254 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:53.410031 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.411593 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.413564 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.414475 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.414622 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.418104 (+ 3482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:53.418119 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:53.418141 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:53.418185 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:53.422372 (+ 4187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.423156 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.423164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.423167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.423349 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.423795 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.423799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.423809 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.424335 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:53.424638 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.424974 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.425187 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:41:53.424360 9154 raft_consensus.cc:2229] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:41:53.427232 8830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:53.427315 9154 raft_consensus.cc:2258] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:41:53.433178 9156 ts_tablet_manager.cc:1918] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:53.433347 9156 log.cc:1196] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:41:53.433799 9156 ts_tablet_manager.cc:1939] T 6559a47b8c1544749a25c66fe72e8d6b P 1d3a9ce1a6fd48a6a970666dec268aab: Deleting consensus metadata
I20260806 20:41:53.435081 9155 ts_tablet_manager.cc:1918] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:41:53.435257 9155 log.cc:1196] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:41:53.435593 8627 catalog_manager.cc:4841] TS 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): tablet 6559a47b8c1544749a25c66fe72e8d6b (table test-table [id=10acad3baea14387a37e34bdc4bfb5e4]) successfully deleted
I20260806 20:41:53.435837 9155 ts_tablet_manager.cc:1939] T 6559a47b8c1544749a25c66fe72e8d6b P 315c5a139a3c4b718a9826c98f83cbdf: Deleting consensus metadata
I20260806 20:41:53.437518 8628 catalog_manager.cc:4841] TS 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): tablet 6559a47b8c1544749a25c66fe72e8d6b (table test-table [id=10acad3baea14387a37e34bdc4bfb5e4]) successfully deleted
I20260806 20:41:53.521243 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8680
I20260806 20:41:53.540355 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8840
W20260806 20:41:53.540304 8629 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:39649 (error 108)
I20260806 20:41:53.542397 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.541868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36028)
0806 20:41:53.542068 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.542190 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260806 20:41:53.542654 8629 catalog_manager.cc:4584] TS 92d2365e9b614ee6b95cf8716741a4a5 (127.23.200.1:39649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6559a47b8c1544749a25c66fe72e8d6b: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
I20260806 20:41:53.556859 9126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.541527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48308 (local address 127.23.200.2:33059)
0806 20:41:53.541719 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.541723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.541741 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:41:53.541795 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.541799 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.541889 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.542022 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:53.543913 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.544748 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.546784 (+ 2036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.547229 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.547323 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.550752 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:53.550764 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:53.550779 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:53.550815 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:53.553933 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.554844 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.554854 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.554857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.555022 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.555415 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.555418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.555420 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.555776 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:53.556034 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.556357 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.556529 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:41:53.557626 8969 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:53.596470 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.595977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36038)
0806 20:41:53.596203 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.596296 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 20:41:53.597560 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.597221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36040)
0806 20:41:53.597367 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.597451 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260806 20:41:53.658097 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.657606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36056)
0806 20:41:53.657829 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.657930 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 20:41:53.659188 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.658869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36064)
0806 20:41:53.659000 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.659072 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 20:41:53.674703 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8840
I20260806 20:41:53.694746 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8977
I20260806 20:41:53.708492 9128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.695601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35050 (local address 127.23.200.3:37317)
0806 20:41:53.695852 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 20:41:53.695857 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:41:53.695882 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:41:53.695941 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:41:53.695945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:41:53.696008 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:41:53.696098 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:41:53.697455 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.698208 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.699705 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:41:53.700104 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:41:53.700202 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:41:53.702822 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:41:53.702836 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:41:53.702853 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:41:53.702898 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:41:53.705718 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.706556 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.706563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.706566 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.706709 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:41:53.707088 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:41:53.707091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:41:53.707093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:41:53.707443 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:41:53.707668 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 20:41:53.708038 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 20:41:53.708203 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260806 20:41:53.709257 9108 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:53.740078 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.739363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36076)
0806 20:41:53.739703 (+ 340us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.739845 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 20:41:53.741379 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.740842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36084)
0806 20:41:53.741102 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.741224 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260806 20:41:53.801713 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8977
I20260806 20:41:53.819607 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 8608
I20260806 20:41:53.820847 8673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:41:53.936471 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 8608
I20260806 20:41:53.936527 9124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:41:53.935503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39649 (local address 127.0.0.1:36088)
0806 20:41:53.936227 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 20:41:53.936343 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
I20260806 20:42:13.959656 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45869
--webserver_interface=127.23.200.62
--webserver_port=44493
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:45869
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:14.301304 9163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:14.301599 9163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:14.301766 9163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:14.308974 9163 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 20:42:14.309072 9163 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 20:42:14.309146 9163 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 20:42:14.309227 9163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:14.309326 9163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:14.309445 9163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:14.309530 9163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:14.309616 9163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:14.319556 9163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:45869
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=44493
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:14.320909 9163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:14.322290 9163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:14.335095 9169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:14.335134 9170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:14.335165 9172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:14.335703 9163 server_base.cc:1034] running on GCE node
I20260806 20:42:14.336694 9163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:14.338047 9163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:14.339264 9163 hybrid_clock.cc:648] HybridClock initialized: now 1786048934339179 us; error 55 us; skew 500 ppm
I20260806 20:42:14.339607 9163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:14 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048934, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:14.343168 9163 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:42:14.344601 9163 webserver.cc:460] Webserver started at http://127.23.200.62:44493/ using document root <none> and password file <none>
I20260806 20:42:14.345302 9163 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:14.345398 9163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:14.350817 9163 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:42:14.353554 9179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:14.354959 9163 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:14.355139 9163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "fd1d4d6e9f5249928d329c78a7464f02"
format_stamp: "Formatted at 2026-08-06 20:41:28 on dist-test-slave-tm1g"
server_key: "1434dc336f9d4821256cea2bf90e3bc7"
server_key_iv: "f47ab5fa85581688763ab1eef6612b84"
server_key_version: "encryptionkey@0"
I20260806 20:42:14.356098 9163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:14.370810 9163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:14.374819 9163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:14.375106 9163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:14.393930 9163 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:45869
I20260806 20:42:14.393965 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:45869 every 8 connection(s)
I20260806 20:42:14.395475 9163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:14.402992 9234 sys_catalog.cc:263] Verifying existing consensus state
I20260806 20:42:14.406168 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 9163
I20260806 20:42:14.406752 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap starting.
I20260806 20:42:14.407380 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:39649
--local_ip_for_outbound_sockets=127.23.200.1
--tserver_master_addrs=127.23.200.62:45869
--webserver_port=41157
--webserver_interface=127.23.200.1
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:42:14.431740 9234 log.cc:824] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:14.448611 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:42:14.449287 9234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Bootstrap complete.
I20260806 20:42:14.458595 9234 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:42:14.459700 9234 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd1d4d6e9f5249928d329c78a7464f02, State: Initialized, Role: FOLLOWER
I20260806 20:42:14.460441 9234 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:42:14.460727 9234 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:14.460877 9234 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:14.461082 9234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 2 FOLLOWER]: Advancing to term 3
I20260806 20:42:14.465027 9234 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:42:14.465576 9234 leader_election.cc:302] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd1d4d6e9f5249928d329c78a7464f02; no voters:
I20260806 20:42:14.466291 9234 leader_election.cc:288] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 20:42:14.466468 9238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 3 FOLLOWER]: Leader election won for term 3
I20260806 20:42:14.467716 9238 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [term 3 LEADER]: Becoming Leader. State: Replica: fd1d4d6e9f5249928d329c78a7464f02, State: Running, Role: LEADER
I20260806 20:42:14.468293 9238 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } }
I20260806 20:42:14.468642 9234 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:14.475101 9240 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd1d4d6e9f5249928d329c78a7464f02. Latest consensus state: current_term: 3 leader_uuid: "fd1d4d6e9f5249928d329c78a7464f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } } }
I20260806 20:42:14.475430 9240 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:14.474710 9239 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd1d4d6e9f5249928d329c78a7464f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd1d4d6e9f5249928d329c78a7464f02" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45869 } } }
I20260806 20:42:14.476003 9239 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:14.487846 9249 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:14.496937 9249 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:14.498546 9249 catalog_manager.cc:1259] Loaded cluster ID: 8cf08da2847a45959ac98fc46c8102fc
I20260806 20:42:14.498667 9249 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:14.501480 9249 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:14.502735 9249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Loaded TSK: 0
I20260806 20:42:14.502880 9249 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd1d4d6e9f5249928d329c78a7464f02: Loaded TSK: 1
F20260806 20:42:14.503688 9249 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f324f66b9ad google::LogMessage::Fail() at ??:0
@ 0x7f324f66e88a google::LogMessage::SendToLog() at ??:0
@ 0x7f324f66b4de google::LogMessage::Flush() at ??:0
@ 0x7f324f66ca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f32533cf7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f32533be73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f326509e1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f326509d783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f32651292a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f32651290c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f32651735e9 std::function<>::operator()() at ??:0
@ 0x7f32650a1cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f326509fc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3265127610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f3265127439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f32522b7345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f32522ba91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f32522ba799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f3252295eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f325556f6db start_thread at ??:0
@ 0x7f324e50561f clone at ??:0
W20260806 20:42:14.787418 9236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:14.787765 9236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:14.787940 9236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:14.795380 9236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:14.795531 9236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:14.795722 9236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:14.805529 9236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:39649
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=41157
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:14.806887 9236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:14.808228 9236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:14.821543 9285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:14.821583 9283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:14.821471 9282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:14.822432 9236 server_base.cc:1034] running on GCE node
I20260806 20:42:14.823051 9236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:14.824023 9236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:14.825261 9236 hybrid_clock.cc:648] HybridClock initialized: now 1786048934825210 us; error 32 us; skew 500 ppm
I20260806 20:42:14.825659 9236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:14 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048934, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:14.828938 9236 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:42:14.830549 9236 webserver.cc:460] Webserver started at http://127.23.200.1:41157/ using document root <none> and password file <none>
I20260806 20:42:14.831262 9236 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:14.831427 9236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:14.836802 9236 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 20:42:14.839671 9292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:14.840937 9236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:14.841115 9236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "92d2365e9b614ee6b95cf8716741a4a5"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "5f180a46d5c2e2a79cbe547017116462"
server_key_iv: "60f37531562cb345afe32f27027941e8"
server_key_version: "encryptionkey@0"
I20260806 20:42:14.842095 9236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:14.863169 9236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:14.867034 9236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:14.867312 9236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:14.868274 9236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:14.870013 9300 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 20:42:14.875669 9236 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 20:42:14.875808 9236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260806 20:42:14.875942 9236 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 20:42:14.879231 9236 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 20:42:14.879333 9236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:14.879662 9300 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Bootstrap starting.
I20260806 20:42:14.907085 9300 log.cc:824] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:14.920885 9300 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 20:42:14.921380 9236 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:39649
I20260806 20:42:14.921427 9413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:39649 every 8 connection(s)
I20260806 20:42:14.921584 9300 tablet_bootstrap.cc:492] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Bootstrap complete.
I20260806 20:42:14.922719 9236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:14.923007 9300 ts_tablet_manager.cc:1397] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Time spent bootstrapping tablet: real 0.044s user 0.027s sys 0.015s
I20260806 20:42:14.928153 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:14.926131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:39469)
0806 20:42:14.927013 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:14.927283 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":411,"threads_started":1}
I20260806 20:42:14.928225 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 9236
W20260806 20:42:14.928774 9414 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:45869 (0 consecutive failures): Network error: Failed to ping master at 127.23.200.62:45869: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
I20260806 20:42:14.929021 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:33059
--local_ip_for_outbound_sockets=127.23.200.2
--tserver_master_addrs=127.23.200.62:45869
--webserver_port=41389
--webserver_interface=127.23.200.2
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:42:14.929689 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:14.929338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:36225)
0806 20:42:14.929506 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:14.929583 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260806 20:42:14.930608 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:14.930210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:55717)
0806 20:42:14.930376 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:14.930470 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 20:42:14.930858 9414 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 20:42:14.933636 9300 raft_consensus.cc:348] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:14.935053 9300 raft_consensus.cc:729] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92d2365e9b614ee6b95cf8716741a4a5, State: Initialized, Role: FOLLOWER
I20260806 20:42:14.935727 9300 consensus_queue.cc:260] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:14.937206 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:14.936884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:36367)
0806 20:42:14.937031 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:14.937103 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 20:42:14.937381 9300 ts_tablet_manager.cc:1428] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5: Time spent starting tablet: real 0.014s user 0.009s sys 0.005s
I20260806 20:42:14.938400 9416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:14.937924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:33415)
0806 20:42:14.938115 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:14.938207 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260806 20:42:15.260695 9418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:15.261013 9418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:15.261185 9418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:15.268260 9418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:15.268420 9418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:15.268594 9418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:15.277961 9418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:33059
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=41389
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:15.279261 9418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:15.280514 9418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:15.293628 9424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:15.293531 9425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:15.293562 9427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:15.294395 9418 server_base.cc:1034] running on GCE node
I20260806 20:42:15.294998 9418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:15.295934 9418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:15.297161 9418 hybrid_clock.cc:648] HybridClock initialized: now 1786048935297129 us; error 45 us; skew 500 ppm
I20260806 20:42:15.297479 9418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:15 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048935, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:15.300516 9418 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:42:15.301903 9418 webserver.cc:460] Webserver started at http://127.23.200.2:41389/ using document root <none> and password file <none>
I20260806 20:42:15.302657 9418 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:15.302803 9418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:15.307991 9418 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 20:42:15.310592 9434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:15.311801 9418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:15.311975 9418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "315c5a139a3c4b718a9826c98f83cbdf"
format_stamp: "Formatted at 2026-08-06 20:41:29 on dist-test-slave-tm1g"
server_key: "b4de5d11143a868355ce652d3b4fecdb"
server_key_iv: "fae544854574a8d446d08bfca2ec0ca0"
server_key_version: "encryptionkey@0"
I20260806 20:42:15.312935 9418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:15.334955 9418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:15.338738 9418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:15.339059 9418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:15.339999 9418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:15.341239 9418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:15.341333 9418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:15.341414 9418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:15.341478 9418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:15.374853 9418 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:33059
I20260806 20:42:15.374943 9550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:33059 every 8 connection(s)
I20260806 20:42:15.376283 9418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:15.377270 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 9418
I20260806 20:42:15.378050 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:37317
--local_ip_for_outbound_sockets=127.23.200.3
--tserver_master_addrs=127.23.200.62:45869
--webserver_port=36605
--webserver_interface=127.23.200.3
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:42:15.381433 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.379418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:53141)
0806 20:42:15.380263 (+ 845us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.380581 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":347,"threads_started":1}
W20260806 20:42:15.382175 9551 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:45869 (0 consecutive failures): Network error: Failed to ping master at 127.23.200.62:45869: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
I20260806 20:42:15.383030 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.382695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:47001)
0806 20:42:15.382847 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.382925 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 20:42:15.383867 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.383546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:37699)
0806 20:42:15.383681 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.383765 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 20:42:15.384152 9551 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 20:42:15.707464 9555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:15.707720 9555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:15.707914 9555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:15.715087 9555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:15.715184 9555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:15.715350 9555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:15.725008 9555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:37317
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=36605
--tserver_master_addrs=127.23.200.62:45869
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:15.726226 9555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:15.727528 9555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:15.740115 9560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:15.740149 9563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:15.741067 9561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:15.741459 9555 server_base.cc:1034] running on GCE node
I20260806 20:42:15.742228 9555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:15.743494 9555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:15.744746 9555 hybrid_clock.cc:648] HybridClock initialized: now 1786048935744682 us; error 49 us; skew 500 ppm
I20260806 20:42:15.745116 9555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:15 dist-test-slave-tm1g krb5kdc[8039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048935, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:15.749188 9555 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:42:15.750887 9555 webserver.cc:460] Webserver started at http://127.23.200.3:36605/ using document root <none> and password file <none>
I20260806 20:42:15.751598 9555 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:15.751742 9555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:15.756831 9555 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260806 20:42:15.759423 9570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:15.760607 9555 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:15.760782 9555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "1d3a9ce1a6fd48a6a970666dec268aab"
format_stamp: "Formatted at 2026-08-06 20:41:30 on dist-test-slave-tm1g"
server_key: "95d9305eab4ee0ae468cdf6ee625a839"
server_key_iv: "864adff332f253f1a47b5cd7d98f6178"
server_key_version: "encryptionkey@0"
I20260806 20:42:15.761811 9555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:15.775446 9555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:15.779604 9555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:15.779878 9555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:15.780773 9555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:15.782051 9555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:15.782167 9555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:15.782296 9555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:15.782450 9555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 20:42:15.815675 9555 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:37317
I20260806 20:42:15.815778 9688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:37317 every 8 connection(s)
I20260806 20:42:15.817251 9555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:15.821861 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.820161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:38449)
0806 20:42:15.820845 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.821080 (+ 235us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":267,"threads_started":1}
W20260806 20:42:15.822590 9689 heartbeater.cc:644] Failed to heartbeat to 127.23.200.62:45869 (0 consecutive failures): Network error: Failed to ping master at 127.23.200.62:45869: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
I20260806 20:42:15.822574 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 9555
I20260806 20:42:15.823509 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.823110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:41721)
0806 20:42:15.823277 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.823370 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 20:42:15.824420 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.824121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:54899)
0806 20:42:15.824249 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.824318 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 20:42:15.824745 9689 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 20:42:15.940147 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:15.939222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:37319)
0806 20:42:15.939856 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:15.939953 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":425,"threads_started":1}
I20260806 20:42:16.191756 9694 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:16.192059 9694 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:16.193710 9694 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
Aug 06 20:42:16 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048934, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
Aug 06 20:42:16 dist-test-slave-tm1g krb5kdc[8039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048934, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:42:16.218819 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.194033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:33059 (local address 127.23.200.1:56761)
0806 20:42:16.194294 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:16.194378 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:16.195023 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:16.199136 (+ 4113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:16.199144 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:16.199714 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:16.200338 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:16.200356 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.201932 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.201936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:16.203255 (+ 1319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:16.203270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.204309 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.204317 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:16.204507 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:16.209334 (+ 4827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:16.209376 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:16.212216 (+ 2840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:16.215679 (+ 3463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:16.215685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:16.215688 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:16.216144 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:16.216558 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:16.216567 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:16.216570 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:16.216702 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:16.217528 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:16.217532 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:16.217699 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 20:42:16.217876 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 20:42:16.218401 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":54,"spinlock_wait_cycles":2816}
I20260806 20:42:16.218982 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.194341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:56761 (local address 127.23.200.2:33059)
0806 20:42:16.195823 (+ 1482us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:16.195839 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:16.195884 (+ 45us) server_negotiation.cc:407] Connection header received
0806 20:42:16.198141 (+ 2257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:16.198154 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:16.198666 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:16.199024 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:16.200550 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.201726 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.203405 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.204038 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.204263 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:16.212471 (+ 8208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:16.212478 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:16.212488 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:16.212526 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:16.215530 (+ 3004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:16.216274 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:16.216279 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:16.216284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:16.216443 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:16.216823 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:16.216827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:16.216829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:16.217397 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:16.217582 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:16.218119 (+ 537us) server_negotiation.cc:299] Negotiation successful
0806 20:42:16.218564 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":1189,"threads_started":1}
I20260806 20:42:16.219267 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.194414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:51175 (local address 127.23.200.3:37317)
0806 20:42:16.196854 (+ 2440us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:16.196867 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:16.196906 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:42:16.199049 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:16.199059 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:16.199469 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:16.199799 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:16.200864 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.202018 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.203442 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.204039 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.204246 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:16.212222 (+ 7976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:16.212230 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:16.212242 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:16.212275 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:16.215126 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:16.215873 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:16.215880 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:16.215885 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:16.216032 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:16.216524 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:16.216529 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:16.216530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:16.217078 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:16.217270 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:16.218503 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0806 20:42:16.218865 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2264,"thread_start_us":2089,"threads_started":1}
I20260806 20:42:16.219739 9695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.194125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:37317 (local address 127.23.200.1:51175)
0806 20:42:16.196136 (+ 2011us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:16.196155 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:16.196271 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:16.199766 (+ 3495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:16.199771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:16.200382 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:16.200689 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:16.200699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.202189 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.202193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:16.203300 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:16.203311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:16.204246 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:16.204253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:16.204503 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:16.209376 (+ 4873us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:16.209397 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:16.211952 (+ 2555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:16.215276 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:16.215290 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:16.215303 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:16.215712 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:16.216148 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:16.216155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:16.216158 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:16.216401 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:16.217210 (+ 809us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:16.217217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:16.217551 (+ 334us) client_negotiation.cc:769] Sending connection context
0806 20:42:16.217872 (+ 321us) client_negotiation.cc:240] Negotiation successful
0806 20:42:16.219336 (+ 1464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1895,"mutex_wait_us":1046,"spinlock_wait_cycles":1536,"thread_start_us":1725,"threads_started":1}
W20260806 20:42:16.221987 9296 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
W20260806 20:42:16.222296 9295 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:42:16.222472 9295 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92d2365e9b614ee6b95cf8716741a4a5; no voters: 1d3a9ce1a6fd48a6a970666dec268aab, 315c5a139a3c4b718a9826c98f83cbdf
I20260806 20:42:16.222939 9694 raft_consensus.cc:2726] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:42:16.385429 9698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.384607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:35681)
0806 20:42:16.385042 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:16.385190 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":204,"threads_started":1}
I20260806 20:42:16.826007 9699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.825212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:50055)
0806 20:42:16.825702 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:16.825810 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":274,"threads_started":1}
I20260806 20:42:16.941802 9700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:16.940998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:45343)
0806 20:42:16.941510 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:16.941590 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":303,"threads_started":1}
I20260806 20:42:17.387043 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:17.386283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:57609)
0806 20:42:17.386809 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:17.386889 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":272,"threads_started":1}
I20260806 20:42:17.827603 9702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:17.826810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:36963)
0806 20:42:17.827312 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:17.827412 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":260,"threads_started":1}
I20260806 20:42:17.943559 9703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:17.942770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:38385)
0806 20:42:17.943283 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:17.943367 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":243,"threads_started":1}
I20260806 20:42:18.205314 9704 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:18.205544 9704 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:18.206229 9704 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
W20260806 20:42:18.207414 9295 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
W20260806 20:42:18.207608 9296 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:42:18.207764 9296 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92d2365e9b614ee6b95cf8716741a4a5; no voters: 1d3a9ce1a6fd48a6a970666dec268aab, 315c5a139a3c4b718a9826c98f83cbdf
I20260806 20:42:18.208016 9704 raft_consensus.cc:2726] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:42:18.388530 9705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:18.387858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:51383)
0806 20:42:18.388287 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:18.388374 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":210,"threads_started":1}
I20260806 20:42:18.829067 9706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:18.828395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:51703)
0806 20:42:18.828826 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:18.828912 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":257,"threads_started":1}
I20260806 20:42:18.945237 9707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:18.944461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:50271)
0806 20:42:18.944921 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:18.945053 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":245,"threads_started":1}
I20260806 20:42:19.390130 9708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:19.389473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:58019)
0806 20:42:19.389874 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:19.389951 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":204,"threads_started":1}
I20260806 20:42:19.830791 9709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:19.829948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:36895)
0806 20:42:19.830522 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:19.830607 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":341,"threads_started":1}
I20260806 20:42:19.947080 9710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:19.946241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:55133)
0806 20:42:19.946809 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:19.946900 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":356,"threads_started":1}
I20260806 20:42:19.954615 9711 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:19.954821 9711 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:19.955520 9711 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
W20260806 20:42:19.956530 9296 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
W20260806 20:42:19.956715 9295 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:42:19.956836 9295 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92d2365e9b614ee6b95cf8716741a4a5; no voters: 1d3a9ce1a6fd48a6a970666dec268aab, 315c5a139a3c4b718a9826c98f83cbdf
I20260806 20:42:19.957057 9711 raft_consensus.cc:2726] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:42:20.391737 9712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:20.390977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:52589)
0806 20:42:20.391502 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:20.391580 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":318,"threads_started":1}
I20260806 20:42:20.832391 9713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:20.831706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:35561)
0806 20:42:20.832147 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:20.832237 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":242,"threads_started":1}
I20260806 20:42:20.948503 9714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:20.947842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:49547)
0806 20:42:20.948256 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:20.948343 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":233,"threads_started":1}
I20260806 20:42:21.393406 9715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:21.392635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:60917)
0806 20:42:21.393146 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:21.393242 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":309,"threads_started":1}
I20260806 20:42:21.834110 9716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:21.833289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:50629)
0806 20:42:21.833738 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:21.833865 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":251,"threads_started":1}
I20260806 20:42:21.950034 9717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:21.949329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:44161)
0806 20:42:21.949743 (+ 414us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:21.949833 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":229,"threads_started":1}
I20260806 20:42:22.395431 9718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:22.394483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:38643)
0806 20:42:22.395099 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:22.395196 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":280,"threads_started":1}
I20260806 20:42:22.835994 9719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:22.835142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:52141)
0806 20:42:22.835675 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:22.835766 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":219,"threads_started":1}
I20260806 20:42:22.951748 9720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:22.951010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:60613)
0806 20:42:22.951493 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:22.951596 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":275,"threads_started":1}
I20260806 20:42:23.397221 9721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:23.396456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:43003)
0806 20:42:23.396941 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:23.397037 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":257,"threads_started":1}
I20260806 20:42:23.622697 9722 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:23.622942 9722 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:23.623713 9722 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
W20260806 20:42:23.624776 9296 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
W20260806 20:42:23.624974 9295 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:42:23.625092 9295 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92d2365e9b614ee6b95cf8716741a4a5; no voters: 1d3a9ce1a6fd48a6a970666dec268aab, 315c5a139a3c4b718a9826c98f83cbdf
I20260806 20:42:23.625339 9722 raft_consensus.cc:2726] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:42:23.837770 9723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:23.837030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:43933)
0806 20:42:23.837484 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:23.837601 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":209,"threads_started":1}
I20260806 20:42:23.953604 9724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:23.952712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:55675)
0806 20:42:23.953255 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:23.953375 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":267,"threads_started":1}
I20260806 20:42:24.398948 9725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:24.398197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:33805)
0806 20:42:24.398700 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:24.398788 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":267,"threads_started":1}
I20260806 20:42:24.839397 9726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:24.838723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:46605)
0806 20:42:24.839162 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:24.839243 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20260806 20:42:24.955508 9727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:24.954675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:43399)
0806 20:42:24.955221 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:24.955300 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":327,"threads_started":1}
I20260806 20:42:25.400619 9728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:25.399855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:60607)
0806 20:42:25.400336 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:25.400442 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":254,"threads_started":1}
I20260806 20:42:25.840935 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:25.840177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:38581)
0806 20:42:25.840677 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:25.840755 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":304,"threads_started":1}
I20260806 20:42:25.956996 9730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:25.956315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:35361)
0806 20:42:25.956742 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:25.956821 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":236,"threads_started":1}
I20260806 20:42:26.402202 9731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:26.401493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:37109)
0806 20:42:26.401954 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:26.402050 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":209,"threads_started":1}
I20260806 20:42:26.842545 9732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:26.841798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:56063)
0806 20:42:26.842269 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:26.842374 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":250,"threads_started":1}
I20260806 20:42:26.958518 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:26.957806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:44023)
0806 20:42:26.958209 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:26.958287 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":211,"threads_started":1}
I20260806 20:42:27.403765 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:27.403031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:43741)
0806 20:42:27.403513 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:27.403594 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":269,"threads_started":1}
I20260806 20:42:27.844131 9735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:27.843417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:49521)
0806 20:42:27.843889 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:27.843967 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":239,"threads_started":1}
I20260806 20:42:27.954732 9736 raft_consensus.cc:482] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:27.954949 9736 raft_consensus.cc:504] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92d2365e9b614ee6b95cf8716741a4a5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39649 } } peers { permanent_uuid: "315c5a139a3c4b718a9826c98f83cbdf" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 33059 } } peers { permanent_uuid: "1d3a9ce1a6fd48a6a970666dec268aab" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 37317 } }
I20260806 20:42:27.955612 9736 leader_election.cc:288] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059), 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317)
W20260806 20:42:27.956665 9295 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1d3a9ce1a6fd48a6a970666dec268aab (127.23.200.3:37317): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
W20260806 20:42:27.956872 9296 leader_election.cc:341] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 315c5a139a3c4b718a9826c98f83cbdf (127.23.200.2:33059): Not found: Tablet not found: 6559a47b8c1544749a25c66fe72e8d6b
I20260806 20:42:27.957033 9296 leader_election.cc:302] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92d2365e9b614ee6b95cf8716741a4a5; no voters: 1d3a9ce1a6fd48a6a970666dec268aab, 315c5a139a3c4b718a9826c98f83cbdf
I20260806 20:42:27.957336 9736 raft_consensus.cc:2726] T 6559a47b8c1544749a25c66fe72e8d6b P 92d2365e9b614ee6b95cf8716741a4a5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 20:42:27.960003 9737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:27.959277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:33367)
0806 20:42:27.959706 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:27.959807 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":243,"threads_started":1}
I20260806 20:42:28.406183 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:28.404821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:60521)
0806 20:42:28.405660 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:28.405857 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":375,"threads_started":1}
I20260806 20:42:28.846021 9739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:28.845191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:50447)
0806 20:42:28.845745 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:28.845859 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":264,"threads_started":1}
I20260806 20:42:28.961712 9740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:28.960819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:49215)
0806 20:42:28.961414 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:28.961505 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":311,"threads_started":1}
I20260806 20:42:29.407931 9741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:29.407193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:40963)
0806 20:42:29.407637 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:29.407725 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":217,"threads_started":1}
I20260806 20:42:29.847839 9742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:29.846960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:38839)
0806 20:42:29.847506 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:29.847587 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":307,"threads_started":1}
I20260806 20:42:29.963902 9743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:29.963064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.1:48963)
0806 20:42:29.963625 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:29.963734 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":341,"threads_started":1}
I20260806 20:42:30.409742 9744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:30.409042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.2:44755)
0806 20:42:30.409466 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:30.409583 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":212,"threads_started":1}
I20260806 20:42:30.827483 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 9236
I20260806 20:42:30.846519 9745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:30.828548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.23.200.1:39649)
0806 20:42:30.829109 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:30.829119 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:30.829154 (+ 35us) server_negotiation.cc:407] Connection header received
0806 20:42:30.829287 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:30.829298 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:30.829634 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:30.829843 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:30.831445 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:30.832369 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:30.833596 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:30.834163 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:30.834261 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:30.841027 (+ 6766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:30.841039 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:30.841055 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:30.841088 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:30.843964 (+ 2876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:30.844567 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:30.844574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:30.844576 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:30.844728 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:30.845031 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:30.845034 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:30.845036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:30.845456 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:30.845693 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:30.845936 (+ 243us) server_negotiation.cc:299] Negotiation successful
0806 20:42:30.846142 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":286,"threads_started":1}
I20260806 20:42:30.847711 9409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:30.849892 9747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:30.849012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:45869 (local address 127.23.200.3:32931)
0806 20:42:30.849564 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:30.849678 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.62:45869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
I20260806 20:42:30.936980 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 9236
I20260806 20:42:30.955856 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 9418
I20260806 20:42:30.969148 9749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:30.956710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58010 (local address 127.23.200.2:33059)
0806 20:42:30.957145 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:30.957150 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:30.957166 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:42:30.957221 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:30.957224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:30.957284 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:30.957376 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:30.958706 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:30.959468 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:30.960610 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:30.961030 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:30.961120 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:30.963737 (+ 2617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:30.963741 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:30.963744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:30.963773 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:30.966842 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:30.967494 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:30.967498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:30.967501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:30.967627 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:30.967956 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:30.967959 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:30.967961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:30.968211 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:30.968407 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:30.968690 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 20:42:30.968850 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":219,"threads_started":1}
I20260806 20:42:30.969904 9546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:31.075572 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 9418
I20260806 20:42:31.096257 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 9555
I20260806 20:42:31.109671 9751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:31.097137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36738 (local address 127.23.200.3:37317)
0806 20:42:31.097730 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:31.097735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:31.097753 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:42:31.097809 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:31.097813 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:31.097882 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:31.097975 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:31.099501 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:31.100299 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:31.101483 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:31.101913 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:31.102007 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:31.104620 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:31.104625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:31.104628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:31.104657 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:31.107403 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:31.107970 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:31.107974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:31.107976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:31.108114 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:31.108420 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:31.108423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:31.108425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:31.108662 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:31.108860 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:31.109075 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 20:42:31.109233 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":335,"threads_started":1}
I20260806 20:42:31.110423 9683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:31.194276 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 9555
2026-08-06T20:42:31Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64251 ms)
[----------] 30 tests from SecurityITest (299674 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:31 dist-test-slave-tm1g krb5kdc[9756](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:31 dist-test-slave-tm1g krb5kdc[9756](info): set up 2 sockets
Aug 06 20:42:31 dist-test-slave-tm1g krb5kdc[9756](info): commencing operation
krb5kdc: starting...
W20260806 20:42:32.322216 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:42:32 dist-test-slave-tm1g krb5kdc[9756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:42:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:42:32.570580 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:41815
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:41815
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:32.909087 9769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:32.909415 9769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:32.909587 9769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:32.916669 9769 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:32.916805 9769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:32.916926 9769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:32.917029 9769 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:32.917115 9769 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:32.926676 9769 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:41815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:41815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:32.927949 9769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:32.929242 9769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:32.942524 9774 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:32.942521 9775 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:32.942562 9777 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:32.943400 9769 server_base.cc:1034] running on GCE node
I20260806 20:42:32.944105 9769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:32.945398 9769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:32.946604 9769 hybrid_clock.cc:648] HybridClock initialized: now 1786048952946542 us; error 54 us; skew 500 ppm
I20260806 20:42:32.946964 9769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:32 dist-test-slave-tm1g krb5kdc[9756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048952, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:32.950142 9769 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:42:32.951584 9769 webserver.cc:460] Webserver started at http://127.23.200.62:39647/ using document root <none> and password file <none>
I20260806 20:42:32.952215 9769 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:32.952325 9769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:32.952580 9769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:32.955715 9769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "8f877f1ab37e486da3d2bce7d419a620"
format_stamp: "Formatted at 2026-08-06 20:42:32 on dist-test-slave-tm1g"
server_key: "4bb977997f826a41e63a5135675632f3"
server_key_iv: "b2bd1d15d3455364fcd338859f6211af"
server_key_version: "encryptionkey@0"
I20260806 20:42:32.956425 9769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "8f877f1ab37e486da3d2bce7d419a620"
format_stamp: "Formatted at 2026-08-06 20:42:32 on dist-test-slave-tm1g"
server_key: "4bb977997f826a41e63a5135675632f3"
server_key_iv: "b2bd1d15d3455364fcd338859f6211af"
server_key_version: "encryptionkey@0"
I20260806 20:42:32.961484 9769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:42:32.964749 9785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:32.966002 9769 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:42:32.966174 9769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "8f877f1ab37e486da3d2bce7d419a620"
format_stamp: "Formatted at 2026-08-06 20:42:32 on dist-test-slave-tm1g"
server_key: "4bb977997f826a41e63a5135675632f3"
server_key_iv: "b2bd1d15d3455364fcd338859f6211af"
server_key_version: "encryptionkey@0"
I20260806 20:42:32.966428 9769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:32.988160 9769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:32.992415 9769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:32.992707 9769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:33.011732 9769 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:41815
I20260806 20:42:33.011765 9837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:41815 every 8 connection(s)
I20260806 20:42:33.013062 9769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:33.013645 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 9769
I20260806 20:42:33.013864 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:33.014716 24352 external_mini_cluster.cc:1428] Setting key 61935db355a8406bcc107b1f4d7c18d9
I20260806 20:42:33.017452 9838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:33.027616 9838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620: Bootstrap starting.
Aug 06 20:42:33 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:33.030917 9838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:33.031940 9838 log.cc:824] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:33.034870 9838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620: No bootstrap required, opened a new log
I20260806 20:42:33.036471 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:33.016655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47830 (local address 127.23.200.62:41815)
0806 20:42:33.017395 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:33.017407 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:33.017450 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:42:33.019218 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:33.019238 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:33.019590 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:33.020013 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:33.021519 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:33.022922 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:33.024236 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:33.024913 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:33.025130 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:33.029197 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:33.029205 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:33.029218 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:33.029253 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:33.032388 (+ 3135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:33.033179 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:33.033185 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:33.033191 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:33.033380 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:33.033771 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:33.033777 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:33.033780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:33.034285 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:33.034661 (+ 376us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:33.035132 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 20:42:33.035553 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":344,"threads_started":1}
I20260806 20:42:33.043154 9838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f877f1ab37e486da3d2bce7d419a620" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41815 } }
I20260806 20:42:33.043635 9838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:33.043763 9838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f877f1ab37e486da3d2bce7d419a620, State: Initialized, Role: FOLLOWER
I20260806 20:42:33.044278 9838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f877f1ab37e486da3d2bce7d419a620" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41815 } }
I20260806 20:42:33.044487 9838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:33.044587 9838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:33.044701 9838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:33.046806 9838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f877f1ab37e486da3d2bce7d419a620" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41815 } }
I20260806 20:42:33.047219 9838 leader_election.cc:302] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f877f1ab37e486da3d2bce7d419a620; no voters:
I20260806 20:42:33.047761 9838 leader_election.cc:288] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:33.047900 9843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:33.048575 9843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [term 1 LEADER]: Becoming Leader. State: Replica: 8f877f1ab37e486da3d2bce7d419a620, State: Running, Role: LEADER
I20260806 20:42:33.049081 9843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f877f1ab37e486da3d2bce7d419a620" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41815 } }
I20260806 20:42:33.049731 9838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:33.052057 9844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f877f1ab37e486da3d2bce7d419a620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f877f1ab37e486da3d2bce7d419a620" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41815 } } }
I20260806 20:42:33.052424 9844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:33.053071 9845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f877f1ab37e486da3d2bce7d419a620. Latest consensus state: current_term: 1 leader_uuid: "8f877f1ab37e486da3d2bce7d419a620" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f877f1ab37e486da3d2bce7d419a620" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 41815 } } }
I20260806 20:42:33.053439 9845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:33.054847 9853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:33.059859 9853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:33.067879 9853 catalog_manager.cc:1347] Generated new cluster ID: ecefedf7245a48d5be3dc3ae30da8f00
I20260806 20:42:33.067992 9853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:33.080628 9853 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:33.082110 9853 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:33.093565 9853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620: Generated new TSK 0
I20260806 20:42:33.094369 9853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:42:33.164479 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41815
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:33.492305 9866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:33.492622 9866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:33.492810 9866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:33.499872 9866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:33.500020 9866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:33.500216 9866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:33.509519 9866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:33.510732 9866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:33.511996 9866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:33.524573 9874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:33.524689 9871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:33.524643 9872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:33.525589 9866 server_base.cc:1034] running on GCE node
I20260806 20:42:33.526151 9866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:33.527119 9866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:33.528390 9866 hybrid_clock.cc:648] HybridClock initialized: now 1786048953528331 us; error 51 us; skew 500 ppm
I20260806 20:42:33.528651 9866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:33 dist-test-slave-tm1g krb5kdc[9756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048953, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:33.531661 9866 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:42:33.532972 9866 webserver.cc:460] Webserver started at http://127.23.200.1:36351/ using document root <none> and password file <none>
I20260806 20:42:33.533641 9866 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:33.533784 9866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:33.534096 9866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:33.537168 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "8c7548538f6a481190b127b95f6157c5"
format_stamp: "Formatted at 2026-08-06 20:42:33 on dist-test-slave-tm1g"
server_key: "94bcfe3585f0c59d661fa2e304dcb989"
server_key_iv: "ceb0d6a6a3b454cc8f07c814894c0771"
server_key_version: "encryptionkey@0"
I20260806 20:42:33.537923 9866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "8c7548538f6a481190b127b95f6157c5"
format_stamp: "Formatted at 2026-08-06 20:42:33 on dist-test-slave-tm1g"
server_key: "94bcfe3585f0c59d661fa2e304dcb989"
server_key_iv: "ceb0d6a6a3b454cc8f07c814894c0771"
server_key_version: "encryptionkey@0"
I20260806 20:42:33.543098 9866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 20:42:33.546294 9881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:33.547572 9866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:42:33.547742 9866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "8c7548538f6a481190b127b95f6157c5"
format_stamp: "Formatted at 2026-08-06 20:42:33 on dist-test-slave-tm1g"
server_key: "94bcfe3585f0c59d661fa2e304dcb989"
server_key_iv: "ceb0d6a6a3b454cc8f07c814894c0771"
server_key_version: "encryptionkey@0"
I20260806 20:42:33.547947 9866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:33.559609 9866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:33.563292 9866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:33.563568 9866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:33.564419 9866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:33.565637 9866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:33.565739 9866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:33.565881 9866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:33.565953 9866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:33.599465 9866 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:42873
I20260806 20:42:33.599524 9998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:42873 every 8 connection(s)
I20260806 20:42:33.600736 9866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:33.610558 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 9866
I20260806 20:42:33.610785 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:42:33.611642 24352 external_mini_cluster.cc:1428] Setting key be96d41fafdaefb74c3588c92ef693a3
Aug 06 20:42:33 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048953, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:33.620661 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:33.603298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:37161 (local address 127.23.200.62:41815)
0806 20:42:33.603730 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:33.603735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:33.604714 (+ 979us) server_negotiation.cc:407] Connection header received
0806 20:42:33.606523 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:33.606527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:33.606594 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:33.606689 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:33.608092 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:33.608794 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:33.610405 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:33.610881 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:33.610982 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:33.615079 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:33.615084 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:33.615087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:33.615117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:33.617621 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:33.618378 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:33.618382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:33.618384 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:33.618505 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:33.618938 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:33.618943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:33.618948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:33.619208 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:33.619366 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:33.620086 (+ 720us) server_negotiation.cc:299] Negotiation successful
0806 20:42:33.620309 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":226,"threads_started":1}
I20260806 20:42:33.621577 10002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:33.603876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41815 (local address 127.23.200.1:37161)
0806 20:42:33.604546 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:33.604586 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:33.606221 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:33.606892 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:33.606899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:33.607394 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:33.607948 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:33.607964 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:33.608923 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:33.608926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:33.610208 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:33.610225 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:33.611117 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:33.611125 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:33.611425 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:33.612305 (+ 880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:33.612340 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:33.614856 (+ 2516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:33.617765 (+ 2909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:33.617774 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:33.617790 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:33.618206 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:33.618620 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:33.618625 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:33.618628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:33.618815 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:33.619340 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:33.619348 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:33.619680 (+ 332us) client_negotiation.cc:769] Sending connection context
0806 20:42:33.619945 (+ 265us) client_negotiation.cc:240] Negotiation successful
0806 20:42:33.620451 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":248,"threads_started":1}
I20260806 20:42:33.623114 9999 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41815
I20260806 20:42:33.623438 9999 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:33.624253 9999 heartbeater.cc:511] Master 127.23.200.62:41815 requested a full tablet report, sending...
I20260806 20:42:33.626631 9803 ts_manager.cc:194] Registered new tserver with Master: 8c7548538f6a481190b127b95f6157c5 (127.23.200.1:42873)
I20260806 20:42:33.628775 9803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:37161
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:42:33.669787 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41815
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:34.000638 10005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:34.000897 10005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:34.000990 10005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:34.008069 10005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:34.008165 10005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:34.008272 10005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:34.017652 10005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:34.018848 10005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:34.020080 10005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:34.033099 10011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:34.033142 10013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:34.033165 10010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:34.034055 10005 server_base.cc:1034] running on GCE node
I20260806 20:42:34.034636 10005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:34.035523 10005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:34.036711 10005 hybrid_clock.cc:648] HybridClock initialized: now 1786048954036664 us; error 50 us; skew 500 ppm
I20260806 20:42:34.037015 10005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:34 dist-test-slave-tm1g krb5kdc[9756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:34.040040 10005 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:42:34.041380 10005 webserver.cc:460] Webserver started at http://127.23.200.2:34279/ using document root <none> and password file <none>
I20260806 20:42:34.042039 10005 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:34.042186 10005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:34.042547 10005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:34.045337 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "bc6a03f00ffe400bb5c35ac58fe436c0"
format_stamp: "Formatted at 2026-08-06 20:42:34 on dist-test-slave-tm1g"
server_key: "ffc4771ea11dfb5383eb42db268cac84"
server_key_iv: "c91401cb688d3d8497edef5e6185869e"
server_key_version: "encryptionkey@0"
I20260806 20:42:34.046087 10005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "bc6a03f00ffe400bb5c35ac58fe436c0"
format_stamp: "Formatted at 2026-08-06 20:42:34 on dist-test-slave-tm1g"
server_key: "ffc4771ea11dfb5383eb42db268cac84"
server_key_iv: "c91401cb688d3d8497edef5e6185869e"
server_key_version: "encryptionkey@0"
I20260806 20:42:34.050998 10005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:42:34.054634 10020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:34.055821 10005 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:42:34.055994 10005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "bc6a03f00ffe400bb5c35ac58fe436c0"
format_stamp: "Formatted at 2026-08-06 20:42:34 on dist-test-slave-tm1g"
server_key: "ffc4771ea11dfb5383eb42db268cac84"
server_key_iv: "c91401cb688d3d8497edef5e6185869e"
server_key_version: "encryptionkey@0"
I20260806 20:42:34.056188 10005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:34.076423 10005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:34.080209 10005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:34.080493 10005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:34.081427 10005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:34.082743 10005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:34.082870 10005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:34.083020 10005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:34.083103 10005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:34.115518 10005 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:38927
I20260806 20:42:34.115604 10138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:38927 every 8 connection(s)
I20260806 20:42:34.116835 10005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:34.123629 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10005
I20260806 20:42:34.123829 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:42:34.124325 24352 external_mini_cluster.cc:1428] Setting key d5ee5d348b37d179a9c168f10ca686ae
Aug 06 20:42:34 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:34.138022 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.119438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:54425 (local address 127.23.200.62:41815)
0806 20:42:34.119617 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.119621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.120986 (+ 1365us) server_negotiation.cc:407] Connection header received
0806 20:42:34.122832 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.122838 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.122929 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.123042 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:34.124662 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.125583 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.127281 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.127724 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.127861 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.131187 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:34.131196 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:34.131203 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:34.131246 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:34.134970 (+ 3724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.135790 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.135795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.135797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.135924 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.136356 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.136361 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.136370 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.136626 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:34.136786 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.137593 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.137773 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 20:42:34.139503 10141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.119978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41815 (local address 127.23.200.2:54425)
0806 20:42:34.120725 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.120772 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.122572 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.123263 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.123271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.123834 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:34.124489 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.124508 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.125733 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.125740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.127131 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.127143 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.127956 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.127963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.128181 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.128843 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:34.128869 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:34.130950 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:34.135112 (+ 4162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:34.135122 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:34.135138 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:34.135648 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:34.136041 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:34.136049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:34.136053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:34.136230 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:34.136758 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:34.136765 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:34.137200 (+ 435us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.137492 (+ 292us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.138056 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":288,"threads_started":1}
I20260806 20:42:34.140959 10139 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41815
I20260806 20:42:34.141258 10139 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:34.142098 10139 heartbeater.cc:511] Master 127.23.200.62:41815 requested a full tablet report, sending...
I20260806 20:42:34.143682 9803 ts_manager.cc:194] Registered new tserver with Master: bc6a03f00ffe400bb5c35ac58fe436c0 (127.23.200.2:38927)
I20260806 20:42:34.145099 9803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:54425
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:42:34.185386 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41815
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:34.511807 10144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:34.512143 10144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:34.512326 10144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:34.519308 10144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:34.519454 10144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:34.519640 10144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:34.528954 10144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:41815
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:34.530238 10144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:34.531562 10144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:34.544898 10149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:34.544847 10152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:34.545338 10150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:34.545840 10144 server_base.cc:1034] running on GCE node
I20260806 20:42:34.546496 10144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:34.547392 10144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:34.548623 10144 hybrid_clock.cc:648] HybridClock initialized: now 1786048954548585 us; error 53 us; skew 500 ppm
I20260806 20:42:34.548929 10144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:34 dist-test-slave-tm1g krb5kdc[9756](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:34.552160 10144 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:42:34.553828 10144 webserver.cc:460] Webserver started at http://127.23.200.3:32907/ using document root <none> and password file <none>
I20260806 20:42:34.554661 10144 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:34.554795 10144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:34.555117 10144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:34.557912 10144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "06d701d94ee047c7a20bd3cc33767959"
format_stamp: "Formatted at 2026-08-06 20:42:34 on dist-test-slave-tm1g"
server_key: "bbd8d1cce7f48776fa92a0dbcdab5ba4"
server_key_iv: "50b50b7a1f7c202655c45637051ac38b"
server_key_version: "encryptionkey@0"
I20260806 20:42:34.558712 10144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "06d701d94ee047c7a20bd3cc33767959"
format_stamp: "Formatted at 2026-08-06 20:42:34 on dist-test-slave-tm1g"
server_key: "bbd8d1cce7f48776fa92a0dbcdab5ba4"
server_key_iv: "50b50b7a1f7c202655c45637051ac38b"
server_key_version: "encryptionkey@0"
I20260806 20:42:34.563982 10144 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:42:34.567387 10159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:34.568641 10144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:42:34.568822 10144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "06d701d94ee047c7a20bd3cc33767959"
format_stamp: "Formatted at 2026-08-06 20:42:34 on dist-test-slave-tm1g"
server_key: "bbd8d1cce7f48776fa92a0dbcdab5ba4"
server_key_iv: "50b50b7a1f7c202655c45637051ac38b"
server_key_version: "encryptionkey@0"
I20260806 20:42:34.569026 10144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:34.591396 10144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:34.595211 10144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:34.595451 10144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:34.596367 10144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:34.597620 10144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:34.597697 10144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:34.597779 10144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:34.597823 10144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:34.632335 10144 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:40047
I20260806 20:42:34.632409 10276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:40047 every 8 connection(s)
I20260806 20:42:34.632527 9999 heartbeater.cc:503] Master 127.23.200.62:41815 was elected leader, sending a full tablet report...
I20260806 20:42:34.633908 10144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:34.640071 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10144
I20260806 20:42:34.640282 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:42:34.640805 24352 external_mini_cluster.cc:1428] Setting key 91f2fbe6cddead5cd0b88af1e781718e
Aug 06 20:42:34 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048954, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:34.654445 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.636595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:36605 (local address 127.23.200.62:41815)
0806 20:42:34.636780 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.636785 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.637966 (+ 1181us) server_negotiation.cc:407] Connection header received
0806 20:42:34.639829 (+ 1863us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.639836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.639940 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.640052 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:34.641882 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.643084 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.644845 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.645236 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.645370 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.648561 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:34.648566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:34.648569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:34.648599 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:34.651362 (+ 2763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.652049 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.652052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.652054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.652214 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.652625 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.652632 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.652636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.652941 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:34.653067 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.653874 (+ 807us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.654032 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:42:34.655567 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.637108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:41815 (local address 127.23.200.3:36605)
0806 20:42:34.637810 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.637847 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.639568 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.640315 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.640330 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.640987 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:34.641704 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.641732 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.643241 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.643247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.644716 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.644729 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.645464 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.645469 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.645652 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.646252 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:34.646273 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:34.648386 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:34.651506 (+ 3120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:34.651516 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:34.651531 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:34.651946 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:34.652307 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:34.652310 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:34.652313 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:34.652496 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:34.653101 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:34.653107 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:34.653490 (+ 383us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.653764 (+ 274us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.654215 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":273,"threads_started":1}
I20260806 20:42:34.656915 10277 heartbeater.cc:347] Connected to a master server at 127.23.200.62:41815
I20260806 20:42:34.657227 10277 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:34.657919 10277 heartbeater.cc:511] Master 127.23.200.62:41815 requested a full tablet report, sending...
I20260806 20:42:34.659318 9803 ts_manager.cc:194] Registered new tserver with Master: 06d701d94ee047c7a20bd3cc33767959 (127.23.200.3:40047)
I20260806 20:42:34.660153 9803 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:36605
I20260806 20:42:34.668287 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:42:34.684692 10001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.672909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47840 (local address 127.23.200.62:41815)
0806 20:42:34.673096 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.673101 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.673630 (+ 529us) server_negotiation.cc:407] Connection header received
0806 20:42:34.673750 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.673753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.673811 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.673890 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:34.675262 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.675956 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.677182 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.677565 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.677654 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.679921 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:34.679926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:34.679929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:34.679958 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:34.682402 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.683027 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.683030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.683039 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.683167 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.683478 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.683481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.683483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.683751 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:34.683928 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.684190 (+ 262us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.684341 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260806 20:42:34.688661 9803 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:42:34.695508 9803 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:42:34.717204 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.707287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35072 (local address 127.23.200.1:42873)
0806 20:42:34.708647 (+ 1360us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.708656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.708696 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:42:34.708815 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.708826 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.709120 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.709437 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:34.710881 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.712566 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.714664 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.716009 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.716170 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.716415 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.716697 (+ 282us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.716863 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1158,"thread_start_us":518,"threads_started":1}
I20260806 20:42:34.717253 10289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.707122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:42873 (local address 127.0.0.1:35072)
0806 20:42:34.707934 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.707983 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.708142 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.710117 (+ 1975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.710126 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.710161 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:34.710591 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.710605 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.712732 (+ 2127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.712739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.714507 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.714519 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.716249 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.716257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.716479 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.716516 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.716651 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.716845 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":337,"threads_started":1}
I20260806 20:42:34.720602 10294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.709983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:40047 (local address 127.0.0.1:39794)
0806 20:42:34.711586 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.711603 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.711705 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.712309 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.712314 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.712338 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:34.712921 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.712932 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.714949 (+ 2017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.714954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.717342 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.717357 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.719592 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.719598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.719858 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.719998 (+ 140us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.720090 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.720225 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":148,"thread_start_us":1312,"threads_started":1}
I20260806 20:42:34.720639 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.708848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46486 (local address 127.23.200.2:38927)
0806 20:42:34.710775 (+ 1927us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.710785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.711636 (+ 851us) server_negotiation.cc:407] Connection header received
0806 20:42:34.711745 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.711755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.711959 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.712146 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:34.713214 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.714971 (+ 1757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.717582 (+ 2611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.718990 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.719119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.719295 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.720198 (+ 903us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.720358 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1555,"thread_start_us":1580,"threads_started":1}
I20260806 20:42:34.720661 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.708983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.23.200.3:40047)
0806 20:42:34.710393 (+ 1410us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.710404 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.711812 (+ 1408us) server_negotiation.cc:407] Connection header received
0806 20:42:34.711899 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.711905 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.712105 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.712299 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:34.713133 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.714785 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.717554 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.718990 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.719119 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.719360 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.720160 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.720322 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1179,"thread_start_us":971,"threads_started":1}
I20260806 20:42:34.721319 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.708355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:38927 (local address 127.0.0.1:46486)
0806 20:42:34.711306 (+ 2951us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.711331 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.711484 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.712406 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.712412 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.712442 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:34.713046 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.713067 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.715169 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.715180 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.717422 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.717436 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.719538 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.719547 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.719857 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.719998 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.720088 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.720935 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2801,"mutex_wait_us":835,"thread_start_us":2677,"threads_started":1}
I20260806 20:42:34.721446 9930 tablet_service.cc:1467] Processing CreateTablet for tablet e1c29396faca4410954cfa0c8f6e8e0e (DEFAULT_TABLE table=test-table [id=2c7fb67f0c5e4d94a2fef18329c2f352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:34.723148 9930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c29396faca4410954cfa0c8f6e8e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:34.724562 10209 tablet_service.cc:1467] Processing CreateTablet for tablet e1c29396faca4410954cfa0c8f6e8e0e (DEFAULT_TABLE table=test-table [id=2c7fb67f0c5e4d94a2fef18329c2f352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:34.725677 10068 tablet_service.cc:1467] Processing CreateTablet for tablet e1c29396faca4410954cfa0c8f6e8e0e (DEFAULT_TABLE table=test-table [id=2c7fb67f0c5e4d94a2fef18329c2f352]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:34.726029 10209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c29396faca4410954cfa0c8f6e8e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:34.726740 10068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1c29396faca4410954cfa0c8f6e8e0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:34.738000 10295 tablet_bootstrap.cc:492] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Bootstrap starting.
I20260806 20:42:34.740702 10297 tablet_bootstrap.cc:492] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Bootstrap starting.
I20260806 20:42:34.741918 10295 tablet_bootstrap.cc:654] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:34.742993 10296 tablet_bootstrap.cc:492] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Bootstrap starting.
I20260806 20:42:34.743239 10295 log.cc:824] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:34.744345 10297 tablet_bootstrap.cc:654] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:34.745530 10297 log.cc:824] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:34.746312 10296 tablet_bootstrap.cc:654] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:34.747088 10295 tablet_bootstrap.cc:492] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: No bootstrap required, opened a new log
I20260806 20:42:34.747385 10296 log.cc:824] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:34.747411 10295 ts_tablet_manager.cc:1397] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260806 20:42:34.750154 10296 tablet_bootstrap.cc:492] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: No bootstrap required, opened a new log
I20260806 20:42:34.750478 10296 ts_tablet_manager.cc:1397] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 20:42:34.752938 10297 tablet_bootstrap.cc:492] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: No bootstrap required, opened a new log
I20260806 20:42:34.753214 10297 ts_tablet_manager.cc:1397] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260806 20:42:34.758497 10295 raft_consensus.cc:348] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.758643 10296 raft_consensus.cc:348] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.758970 10295 raft_consensus.cc:374] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:34.759063 10295 raft_consensus.cc:729] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc6a03f00ffe400bb5c35ac58fe436c0, State: Initialized, Role: FOLLOWER
I20260806 20:42:34.759124 10296 raft_consensus.cc:374] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:34.759254 10296 raft_consensus.cc:729] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7548538f6a481190b127b95f6157c5, State: Initialized, Role: FOLLOWER
I20260806 20:42:34.759681 10295 consensus_queue.cc:260] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.759914 10296 consensus_queue.cc:260] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.761162 10139 heartbeater.cc:503] Master 127.23.200.62:41815 was elected leader, sending a full tablet report...
I20260806 20:42:34.761477 10295 ts_tablet_manager.cc:1428] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260806 20:42:34.763561 10296 ts_tablet_manager.cc:1428] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Time spent starting tablet: real 0.013s user 0.009s sys 0.002s
I20260806 20:42:34.763917 10297 raft_consensus.cc:348] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.764344 10297 raft_consensus.cc:374] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:34.764464 10297 raft_consensus.cc:729] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d701d94ee047c7a20bd3cc33767959, State: Initialized, Role: FOLLOWER
I20260806 20:42:34.765091 10297 consensus_queue.cc:260] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.766919 10277 heartbeater.cc:503] Master 127.23.200.62:41815 was elected leader, sending a full tablet report...
I20260806 20:42:34.768585 10297 ts_tablet_manager.cc:1428] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260806 20:42:34.847651 10303 raft_consensus.cc:482] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:34.847990 10303 raft_consensus.cc:504] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.849831 10303 leader_election.cc:288] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8c7548538f6a481190b127b95f6157c5 (127.23.200.1:42873), bc6a03f00ffe400bb5c35ac58fe436c0 (127.23.200.2:38927)
W20260806 20:42:34.853139 10000 tablet.cc:2369] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:34.856837 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.850236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:42873 (local address 127.23.200.3:39153)
0806 20:42:34.850487 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.850510 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.850698 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.851337 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.851344 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.851379 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:34.851579 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.851586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.852948 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.852954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.854976 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.854990 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.856316 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.856320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.856451 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.856474 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.856555 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.856641 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 20:42:34.856997 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.850240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:39153 (local address 127.23.200.1:42873)
0806 20:42:34.850491 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.850498 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.850611 (+ 113us) server_negotiation.cc:407] Connection header received
0806 20:42:34.850882 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.850891 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.850991 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.851110 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:34.851707 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.852805 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.855156 (+ 2351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.856167 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.856355 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.856429 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.856607 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.856749 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 20:42:34.857975 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.851109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:46699 (local address 127.23.200.2:38927)
0806 20:42:34.851348 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.851354 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.851599 (+ 245us) server_negotiation.cc:407] Connection header received
0806 20:42:34.851886 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.851892 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.851994 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.852092 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:34.852654 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.853745 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.855836 (+ 2091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.857090 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.857252 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.857320 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.857603 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.857731 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 20:42:34.858008 10304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.850719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:38927 (local address 127.23.200.3:46699)
0806 20:42:34.851489 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:34.851505 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:34.851653 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:34.852110 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:34.852115 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:34.852139 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:34.852471 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.852480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.853937 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.853943 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.855662 (+ 1719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:34.855674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.857328 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.857332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:34.857443 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.857465 (+ 22us) client_negotiation.cc:769] Sending connection context
0806 20:42:34.857573 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 20:42:34.857691 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":423,"threads_started":1}
I20260806 20:42:34.857874 9953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c29396faca4410954cfa0c8f6e8e0e" candidate_uuid: "06d701d94ee047c7a20bd3cc33767959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7548538f6a481190b127b95f6157c5" is_pre_election: true
I20260806 20:42:34.858502 9953 raft_consensus.cc:2454] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d701d94ee047c7a20bd3cc33767959 in term 0.
I20260806 20:42:34.859254 10160 leader_election.cc:302] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d701d94ee047c7a20bd3cc33767959, 8c7548538f6a481190b127b95f6157c5; no voters:
I20260806 20:42:34.859383 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c29396faca4410954cfa0c8f6e8e0e" candidate_uuid: "06d701d94ee047c7a20bd3cc33767959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" is_pre_election: true
I20260806 20:42:34.859683 10303 raft_consensus.cc:2781] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:42:34.859814 10303 raft_consensus.cc:482] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:42:34.859894 10303 raft_consensus.cc:3037] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:34.859944 10089 raft_consensus.cc:2454] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06d701d94ee047c7a20bd3cc33767959 in term 0.
I20260806 20:42:34.862396 10303 raft_consensus.cc:504] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
I20260806 20:42:34.862903 10303 leader_election.cc:288] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [CANDIDATE]: Term 1 election: Requested vote from peers 8c7548538f6a481190b127b95f6157c5 (127.23.200.1:42873), bc6a03f00ffe400bb5c35ac58fe436c0 (127.23.200.2:38927)
I20260806 20:42:34.863346 9953 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c29396faca4410954cfa0c8f6e8e0e" candidate_uuid: "06d701d94ee047c7a20bd3cc33767959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c7548538f6a481190b127b95f6157c5"
I20260806 20:42:34.863396 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1c29396faca4410954cfa0c8f6e8e0e" candidate_uuid: "06d701d94ee047c7a20bd3cc33767959" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0"
I20260806 20:42:34.863533 9953 raft_consensus.cc:3037] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:34.863600 10089 raft_consensus.cc:3037] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:34.865711 9953 raft_consensus.cc:2454] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d701d94ee047c7a20bd3cc33767959 in term 1.
I20260806 20:42:34.865846 10089 raft_consensus.cc:2454] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06d701d94ee047c7a20bd3cc33767959 in term 1.
I20260806 20:42:34.866097 10160 leader_election.cc:302] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06d701d94ee047c7a20bd3cc33767959, 8c7548538f6a481190b127b95f6157c5; no voters:
I20260806 20:42:34.866436 10303 raft_consensus.cc:2781] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:34.866901 10303 raft_consensus.cc:686] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 1 LEADER]: Becoming Leader. State: Replica: 06d701d94ee047c7a20bd3cc33767959, State: Running, Role: LEADER
I20260806 20:42:34.867354 10303 consensus_queue.cc:237] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } }
W20260806 20:42:34.868768 10140 tablet.cc:2369] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:34.872068 9803 catalog_manager.cc:5495] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 reported cstate change: term changed from 0 to 1, leader changed from <none> to 06d701d94ee047c7a20bd3cc33767959 (127.23.200.3). New cstate: current_term: 1 leader_uuid: "06d701d94ee047c7a20bd3cc33767959" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06d701d94ee047c7a20bd3cc33767959" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 40047 } health_report { overall_health: HEALTHY } } }
W20260806 20:42:34.885771 10278 tablet.cc:2369] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:34.907562 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.902750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39804 (local address 127.23.200.3:40047)
0806 20:42:34.902927 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.902931 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.902948 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:42:34.903109 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.903113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.903196 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.903319 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:42:34.903672 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.904350 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.905648 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.906029 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.906116 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.906615 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:42:34.907127 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:42:34.907240 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.907306 (+ 66us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.907369 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:42:34.915416 10089 raft_consensus.cc:1261] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 1 FOLLOWER]: Refusing update from remote peer 06d701d94ee047c7a20bd3cc33767959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:34.915416 9953 raft_consensus.cc:1261] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 1 FOLLOWER]: Refusing update from remote peer 06d701d94ee047c7a20bd3cc33767959: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:34.916365 10305 consensus_queue.cc:1035] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7548538f6a481190b127b95f6157c5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 42873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:34.916720 10303 consensus_queue.cc:1035] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc6a03f00ffe400bb5c35ac58fe436c0" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:34.927736 10308 mvcc.cc:205] Tried to move back new op lower bound from 7315656519324000256 to 7315656519140265984. Current Snapshot: MvccSnapshot[applied={T|T < 7315656519324000256}]
I20260806 20:42:34.928710 10309 mvcc.cc:205] Tried to move back new op lower bound from 7315656519324000256 to 7315656519140265984. Current Snapshot: MvccSnapshot[applied={T|T < 7315656519324000256}]
I20260806 20:42:34.941735 9802 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:34.942011 9802 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:34.945621 9802 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8f877f1ab37e486da3d2bce7d419a620: Sending DeleteTablet for 3 replicas of tablet e1c29396faca4410954cfa0c8f6e8e0e
I20260806 20:42:34.946769 9930 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c29396faca4410954cfa0c8f6e8e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:34 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:35072
I20260806 20:42:34.946884 10068 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c29396faca4410954cfa0c8f6e8e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:34 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:46486
I20260806 20:42:34.947161 10209 tablet_service.cc:1514] Processing DeleteTablet for tablet e1c29396faca4410954cfa0c8f6e8e0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:34 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:39794
I20260806 20:42:34.949426 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 9866
I20260806 20:42:34.957578 10320 tablet_replica.cc:330] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: stopping tablet replica
I20260806 20:42:34.957993 10320 raft_consensus.cc:2229] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:42:34.958779 10320 raft_consensus.cc:2258] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:34.961005 10320 ts_tablet_manager.cc:1905] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:34.966804 10318 tablet_replica.cc:330] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: stopping tablet replica
I20260806 20:42:34.967226 10318 raft_consensus.cc:2229] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:34.967722 10318 raft_consensus.cc:2258] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:34.968506 10319 tablet_replica.cc:330] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: stopping tablet replica
I20260806 20:42:34.968890 10319 raft_consensus.cc:2229] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:34.969357 10319 raft_consensus.cc:2258] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:34.970340 10318 ts_tablet_manager.cc:1905] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:34.971882 10320 ts_tablet_manager.cc:1918] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:34.973901 10319 ts_tablet_manager.cc:1905] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:34.976668 10320 log.cc:1196] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/e1c29396faca4410954cfa0c8f6e8e0e
I20260806 20:42:34.977281 10320 ts_tablet_manager.cc:1939] T e1c29396faca4410954cfa0c8f6e8e0e P 06d701d94ee047c7a20bd3cc33767959: Deleting consensus metadata
I20260806 20:42:34.978082 10318 ts_tablet_manager.cc:1918] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:34.978329 10318 log.cc:1196] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/e1c29396faca4410954cfa0c8f6e8e0e
I20260806 20:42:34.978964 10318 ts_tablet_manager.cc:1939] T e1c29396faca4410954cfa0c8f6e8e0e P bc6a03f00ffe400bb5c35ac58fe436c0: Deleting consensus metadata
I20260806 20:42:34.980865 9786 catalog_manager.cc:4841] TS bc6a03f00ffe400bb5c35ac58fe436c0 (127.23.200.2:38927): tablet e1c29396faca4410954cfa0c8f6e8e0e (table test-table [id=2c7fb67f0c5e4d94a2fef18329c2f352]) successfully deleted
I20260806 20:42:34.981298 10319 ts_tablet_manager.cc:1918] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:34.981455 10319 log.cc:1196] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/e1c29396faca4410954cfa0c8f6e8e0e
Aug 06 20:42:34 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:42:34.981894 9789 catalog_manager.cc:4841] TS 06d701d94ee047c7a20bd3cc33767959 (127.23.200.3:40047): tablet e1c29396faca4410954cfa0c8f6e8e0e (table test-table [id=2c7fb67f0c5e4d94a2fef18329c2f352]) successfully deleted
I20260806 20:42:34.982002 10319 ts_tablet_manager.cc:1939] T e1c29396faca4410954cfa0c8f6e8e0e P 8c7548538f6a481190b127b95f6157c5: Deleting consensus metadata
I20260806 20:42:34.983675 9786 catalog_manager.cc:4841] TS 8c7548538f6a481190b127b95f6157c5 (127.23.200.1:42873): tablet e1c29396faca4410954cfa0c8f6e8e0e (table test-table [id=2c7fb67f0c5e4d94a2fef18329c2f352]) successfully deleted
I20260806 20:42:34.989104 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:34.951554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.23.200.1:42873)
0806 20:42:34.951774 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:34.951783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:34.969232 (+ 17449us) server_negotiation.cc:407] Connection header received
0806 20:42:34.969588 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:34.969594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:34.969688 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:34.969773 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:34.971695 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.972912 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.974905 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:34.975604 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:34.975706 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:34.983080 (+ 7374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:34.983092 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:34.983106 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:34.983139 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:34.986203 (+ 3064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.986976 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.986982 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.986985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.987147 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:34.987553 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:34.987557 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:34.987559 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:34.987892 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:34.988198 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:34.988504 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 20:42:34.988730 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:42:34.989822 9994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:35.093648 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 9866
I20260806 20:42:35.112669 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10005
Aug 06 20:42:35 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:42:35.132665 10292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:35.113626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.23.200.2:38927)
0806 20:42:35.113846 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:35.113853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:35.113875 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:42:35.113945 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:35.113950 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:35.114055 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:35.114172 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:35.115913 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:35.117175 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:35.119260 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:35.119992 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:35.120127 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:35.125451 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:35.125465 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:35.125483 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:35.125523 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:35.129415 (+ 3892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:35.130285 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:35.130294 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:35.130298 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:35.130470 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:35.130916 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:35.130921 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:35.130925 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:35.131393 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:35.131676 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:35.132042 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 20:42:35.132269 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:42:35.133466 10134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:35.224817 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10005
I20260806 20:42:35.243119 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10144
Aug 06 20:42:35 dist-test-slave-tm1g krb5kdc[9756](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048952, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:42:35.260414 10293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:35.243976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.23.200.3:40047)
0806 20:42:35.244159 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:35.244167 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:35.244224 (+ 57us) server_negotiation.cc:407] Connection header received
0806 20:42:35.244447 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:35.244452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:35.244564 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:35.244677 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:35.246108 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:35.247017 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:35.248331 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:35.248815 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:35.248963 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:35.254221 (+ 5258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:35.254234 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:35.254250 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:35.254287 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:35.257446 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:35.258160 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:35.258173 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:35.258175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:35.258385 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:35.258760 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:35.258765 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:35.258768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:35.259150 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:35.259474 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:35.259804 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 20:42:35.260039 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:42:35.261155 10271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:35.386229 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10144
I20260806 20:42:35.407676 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 9769
I20260806 20:42:35.408820 9833 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:35.519503 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 9769
2026-08-06T20:42:35Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4327 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T20:42:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:35Z Disabled control of system clock
I20260806 20:42:35.596973 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:42643
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:42643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 20:42:35.929512 10329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:35.929774 10329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:35.929855 10329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:35.936826 10329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:35.936916 10329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:35.936954 10329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:35.937007 10329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:35.937044 10329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:35.945613 10329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:42643
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:42643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:35.946861 10329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:35.948531 10329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:35.959899 10334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:35.959936 10335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:35.959863 10337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:35.961156 10329 server_base.cc:1034] running on GCE node
I20260806 20:42:35.961804 10329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:35.963140 10329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:35.964339 10329 hybrid_clock.cc:648] HybridClock initialized: now 1786048955964301 us; error 25 us; skew 500 ppm
I20260806 20:42:35.964676 10329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:35.971052 10329 webserver.cc:460] Webserver started at http://127.23.200.62:41413/ using document root <none> and password file <none>
I20260806 20:42:35.971709 10329 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:35.971825 10329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:35.972151 10329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:35.974958 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "6e40d2dc6fb14846a28d72b1b8f5825d"
format_stamp: "Formatted at 2026-08-06 20:42:35 on dist-test-slave-tm1g"
server_key: "2603da83f943462502f6fcc591faacf7"
server_key_iv: "49b19c89130d733aa871a27ce91c8784"
server_key_version: "encryptionkey@0"
I20260806 20:42:35.975715 10329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "6e40d2dc6fb14846a28d72b1b8f5825d"
format_stamp: "Formatted at 2026-08-06 20:42:35 on dist-test-slave-tm1g"
server_key: "2603da83f943462502f6fcc591faacf7"
server_key_iv: "49b19c89130d733aa871a27ce91c8784"
server_key_version: "encryptionkey@0"
I20260806 20:42:35.980607 10329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:42:35.983973 10344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:35.985241 10329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:42:35.985430 10329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "6e40d2dc6fb14846a28d72b1b8f5825d"
format_stamp: "Formatted at 2026-08-06 20:42:35 on dist-test-slave-tm1g"
server_key: "2603da83f943462502f6fcc591faacf7"
server_key_iv: "49b19c89130d733aa871a27ce91c8784"
server_key_version: "encryptionkey@0"
I20260806 20:42:35.985639 10329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:36.003111 10329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:36.003855 10329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:36.004168 10329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:36.022286 10329 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:42643
I20260806 20:42:36.022372 10396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:42643 every 8 connection(s)
I20260806 20:42:36.023597 10329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:36.027243 10397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:36.027714 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10329
I20260806 20:42:36.027889 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:36.028359 24352 external_mini_cluster.cc:1428] Setting key 0c29f0a9d3696c0f28dcd6efbbd086dd
I20260806 20:42:36.042191 10397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d: Bootstrap starting.
I20260806 20:42:36.044209 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:36.030182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43544 (local address 127.23.200.62:42643)
0806 20:42:36.031359 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:36.031386 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:36.031452 (+ 66us) server_negotiation.cc:407] Connection header received
0806 20:42:36.034030 (+ 2578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:36.034065 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:36.034687 (+ 622us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:36.035139 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:36.035667 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:36.037530 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:36.039131 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:36.040191 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:36.040362 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:36.041535 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:36.041548 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:36.041566 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:36.042265 (+ 699us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:36.042374 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:36.042398 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:36.042526 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:36.042681 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 20:42:36.042976 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":633,"thread_start_us":324,"threads_started":1}
I20260806 20:42:36.045719 10397 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:36.046844 10397 log.cc:824] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:36.049286 10397 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d: No bootstrap required, opened a new log
I20260806 20:42:36.056468 10397 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42643 } }
I20260806 20:42:36.056777 10397 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:36.056849 10397 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e40d2dc6fb14846a28d72b1b8f5825d, State: Initialized, Role: FOLLOWER
I20260806 20:42:36.057360 10397 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42643 } }
I20260806 20:42:36.057578 10397 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:36.057673 10397 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:36.057786 10397 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:36.059883 10397 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42643 } }
I20260806 20:42:36.060304 10397 leader_election.cc:302] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6e40d2dc6fb14846a28d72b1b8f5825d; no voters:
I20260806 20:42:36.060784 10397 leader_election.cc:288] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:36.060933 10402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:36.061626 10402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [term 1 LEADER]: Becoming Leader. State: Replica: 6e40d2dc6fb14846a28d72b1b8f5825d, State: Running, Role: LEADER
I20260806 20:42:36.062134 10402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42643 } }
I20260806 20:42:36.062826 10397 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:36.064769 10404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6e40d2dc6fb14846a28d72b1b8f5825d. Latest consensus state: current_term: 1 leader_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42643 } } }
I20260806 20:42:36.065161 10404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:36.065514 10403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6e40d2dc6fb14846a28d72b1b8f5825d" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42643 } } }
I20260806 20:42:36.065868 10403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:36.067252 10408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:36.072561 10408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:36.080413 10408 catalog_manager.cc:1347] Generated new cluster ID: e8b6038b71b14c619cb34260b7b0fb27
I20260806 20:42:36.080523 10408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:36.092968 10408 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:36.094137 10408 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:36.108553 10408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d: Generated new TSK 0
I20260806 20:42:36.109359 10408 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:42:36.113775 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42643
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 20:42:36.460316 10423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:36.460605 10423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:36.460693 10423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:36.468088 10423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:36.468189 10423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:36.468299 10423 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:36.477303 10423 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:36.478652 10423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:36.480428 10423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:36.494443 10428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:36.494501 10429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:36.494410 10431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:36.495615 10423 server_base.cc:1034] running on GCE node
I20260806 20:42:36.496286 10423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:36.497241 10423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:36.498459 10423 hybrid_clock.cc:648] HybridClock initialized: now 1786048956498419 us; error 31 us; skew 500 ppm
I20260806 20:42:36.498880 10423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:36.501118 10423 webserver.cc:460] Webserver started at http://127.23.200.1:36517/ using document root <none> and password file <none>
I20260806 20:42:36.501827 10423 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:36.501956 10423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:36.502342 10423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:36.505218 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "1d1565943089418fb78f8b181ce137de"
format_stamp: "Formatted at 2026-08-06 20:42:36 on dist-test-slave-tm1g"
server_key: "63a48f3b631eec7ac7a4d466bba00962"
server_key_iv: "4ffa1b48991a31228983a24947941c02"
server_key_version: "encryptionkey@0"
I20260806 20:42:36.506052 10423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "1d1565943089418fb78f8b181ce137de"
format_stamp: "Formatted at 2026-08-06 20:42:36 on dist-test-slave-tm1g"
server_key: "63a48f3b631eec7ac7a4d466bba00962"
server_key_iv: "4ffa1b48991a31228983a24947941c02"
server_key_version: "encryptionkey@0"
I20260806 20:42:36.511237 10423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:42:36.514636 10439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:36.515908 10423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:42:36.516093 10423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "1d1565943089418fb78f8b181ce137de"
format_stamp: "Formatted at 2026-08-06 20:42:36 on dist-test-slave-tm1g"
server_key: "63a48f3b631eec7ac7a4d466bba00962"
server_key_iv: "4ffa1b48991a31228983a24947941c02"
server_key_version: "encryptionkey@0"
I20260806 20:42:36.516306 10423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:36.576371 10423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:36.577291 10423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:36.577633 10423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:36.578722 10423 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:36.580210 10423 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:36.580349 10423 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:36.580477 10423 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:36.580576 10423 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:36.613873 10423 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:33125
I20260806 20:42:36.613931 10554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:33125 every 8 connection(s)
I20260806 20:42:36.615334 10423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:36.618103 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10423
I20260806 20:42:36.618299 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:42:36.618866 24352 external_mini_cluster.cc:1428] Setting key 498ea5114934c650ed8efe4c918a2348
I20260806 20:42:36.621910 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42643
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 20:42:36.629221 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:36.617948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:38139 (local address 127.23.200.62:42643)
0806 20:42:36.618531 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:36.618549 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:36.620629 (+ 2080us) server_negotiation.cc:407] Connection header received
0806 20:42:36.621970 (+ 1341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:36.621978 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:36.622072 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:36.622185 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:36.623306 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:36.624472 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:36.626038 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:36.626778 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:36.626928 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:36.628231 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:36.628240 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:36.628247 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:36.628273 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:36.628326 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:36.628345 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:36.628445 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:36.628656 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 20:42:36.628769 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":265,"threads_started":1}
I20260806 20:42:36.629984 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:36.618678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42643 (local address 127.23.200.1:38139)
0806 20:42:36.619759 (+ 1081us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:36.619811 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:36.621679 (+ 1868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:36.622474 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:36.622483 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:36.622575 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:36.623127 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:36.623145 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:36.624639 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:36.624643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:36.625886 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:36.625899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:36.626978 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:36.626983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:36.627193 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:36.627911 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:36.627914 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:36.627954 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:36.628033 (+ 79us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:36.628448 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:36.628456 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:36.628461 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 20:42:36.628607 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 20:42:36.628898 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":355,"threads_started":1}
I20260806 20:42:36.631582 10555 heartbeater.cc:347] Connected to a master server at 127.23.200.62:42643
I20260806 20:42:36.631917 10555 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:36.632792 10555 heartbeater.cc:511] Master 127.23.200.62:42643 requested a full tablet report, sending...
I20260806 20:42:36.634980 10361 ts_manager.cc:194] Registered new tserver with Master: 1d1565943089418fb78f8b181ce137de (127.23.200.1:33125)
I20260806 20:42:36.636469 10361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.1:38139
W20260806 20:42:36.958992 10559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:36.959303 10559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:36.959471 10559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:36.967352 10559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:36.967478 10559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:36.967639 10559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:36.976691 10559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:36.978032 10559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:36.979931 10559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:36.993523 10566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:36.993559 10567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:36.993662 10569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:36.994243 10559 server_base.cc:1034] running on GCE node
I20260806 20:42:36.994917 10559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:36.995823 10559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:36.997027 10559 hybrid_clock.cc:648] HybridClock initialized: now 1786048956996986 us; error 27 us; skew 500 ppm
I20260806 20:42:36.997345 10559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:36.999570 10559 webserver.cc:460] Webserver started at http://127.23.200.2:45843/ using document root <none> and password file <none>
I20260806 20:42:37.000325 10559 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:37.000437 10559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:37.000778 10559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:37.003746 10559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "9f6f85df031b4b1caf35015bdc464141"
format_stamp: "Formatted at 2026-08-06 20:42:36 on dist-test-slave-tm1g"
server_key: "e57d7a1ce47cc3d2ab3308f895fa7093"
server_key_iv: "f836c088674fe0bbc53554a25daac8c8"
server_key_version: "encryptionkey@0"
I20260806 20:42:37.004509 10559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "9f6f85df031b4b1caf35015bdc464141"
format_stamp: "Formatted at 2026-08-06 20:42:36 on dist-test-slave-tm1g"
server_key: "e57d7a1ce47cc3d2ab3308f895fa7093"
server_key_iv: "f836c088674fe0bbc53554a25daac8c8"
server_key_version: "encryptionkey@0"
I20260806 20:42:37.009641 10559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260806 20:42:37.013083 10575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:37.014382 10559 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:42:37.014544 10559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "9f6f85df031b4b1caf35015bdc464141"
format_stamp: "Formatted at 2026-08-06 20:42:36 on dist-test-slave-tm1g"
server_key: "e57d7a1ce47cc3d2ab3308f895fa7093"
server_key_iv: "f836c088674fe0bbc53554a25daac8c8"
server_key_version: "encryptionkey@0"
I20260806 20:42:37.014755 10559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:37.026180 10559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:37.026942 10559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:37.027240 10559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:37.028226 10559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:37.029664 10559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:37.029779 10559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:37.029909 10559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:37.030002 10559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:37.063835 10559 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:35111
I20260806 20:42:37.063881 10690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:35111 every 8 connection(s)
I20260806 20:42:37.065183 10559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:37.070391 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10559
I20260806 20:42:37.070581 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:42:37.071138 24352 external_mini_cluster.cc:1428] Setting key cf575036ce56e9f8811922d2bfd05ab9
I20260806 20:42:37.073985 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42643
--builtin_ntp_servers=127.23.200.20:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 20:42:37.079285 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.067818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:40431 (local address 127.23.200.62:42643)
0806 20:42:37.068024 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.068029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.069401 (+ 1372us) server_negotiation.cc:407] Connection header received
0806 20:42:37.071778 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.071785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.071878 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.072002 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:37.073141 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.073986 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.076506 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.076895 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.076989 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.078571 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:37.078575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:37.078579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:37.078596 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:37.078636 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:37.078647 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:37.078719 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.078972 (+ 253us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.079060 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:42:37.080711 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.068329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42643 (local address 127.23.200.2:40431)
0806 20:42:37.069263 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.069298 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.071460 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.072248 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.072266 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.072348 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:37.072968 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.072998 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.074160 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.074167 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.076367 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.076384 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.077115 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.077123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.077451 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.078203 (+ 752us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:37.078209 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:37.078261 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:37.078387 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:37.078745 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:37.078758 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:37.078768 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.078944 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.079356 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":295,"threads_started":1}
I20260806 20:42:37.082175 10691 heartbeater.cc:347] Connected to a master server at 127.23.200.62:42643
I20260806 20:42:37.082501 10691 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:37.083333 10691 heartbeater.cc:511] Master 127.23.200.62:42643 requested a full tablet report, sending...
I20260806 20:42:37.084924 10361 ts_manager.cc:194] Registered new tserver with Master: 9f6f85df031b4b1caf35015bdc464141 (127.23.200.2:35111)
I20260806 20:42:37.086203 10361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.2:40431
W20260806 20:42:37.401307 10694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:37.401568 10694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:37.401652 10694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:37.408690 10694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:37.408777 10694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:37.408877 10694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:37.417351 10694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:42643
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:37.418707 10694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:37.420799 10694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:37.433369 10703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:37.433552 10701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:37.433581 10700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:37.434648 10694 server_base.cc:1034] running on GCE node
I20260806 20:42:37.435222 10694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:37.436026 10694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:37.437248 10694 hybrid_clock.cc:648] HybridClock initialized: now 1786048957437213 us; error 51 us; skew 500 ppm
I20260806 20:42:37.437575 10694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:37.439734 10694 webserver.cc:460] Webserver started at http://127.23.200.3:40149/ using document root <none> and password file <none>
I20260806 20:42:37.440428 10694 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:37.440555 10694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:37.440862 10694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:37.443596 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "4b616c3761a740b39f6f464d92279d0b"
format_stamp: "Formatted at 2026-08-06 20:42:37 on dist-test-slave-tm1g"
server_key: "aa83885b5ad4409054dbff63afc3f0d3"
server_key_iv: "eafb265122283279ddd7221a05536bdb"
server_key_version: "encryptionkey@0"
I20260806 20:42:37.444350 10694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "4b616c3761a740b39f6f464d92279d0b"
format_stamp: "Formatted at 2026-08-06 20:42:37 on dist-test-slave-tm1g"
server_key: "aa83885b5ad4409054dbff63afc3f0d3"
server_key_iv: "eafb265122283279ddd7221a05536bdb"
server_key_version: "encryptionkey@0"
I20260806 20:42:37.449115 10694 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260806 20:42:37.452350 10710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:37.453522 10694 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 20:42:37.453689 10694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "4b616c3761a740b39f6f464d92279d0b"
format_stamp: "Formatted at 2026-08-06 20:42:37 on dist-test-slave-tm1g"
server_key: "aa83885b5ad4409054dbff63afc3f0d3"
server_key_iv: "eafb265122283279ddd7221a05536bdb"
server_key_version: "encryptionkey@0"
I20260806 20:42:37.453892 10694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:37.489085 10694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:37.490036 10694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:37.490401 10694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:37.491334 10694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:37.492784 10694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:37.492865 10694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:37.492992 10694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:37.493085 10694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:37.526130 10694 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:39515
I20260806 20:42:37.526226 10826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:39515 every 8 connection(s)
I20260806 20:42:37.527401 10694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:37.532440 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10694
I20260806 20:42:37.532640 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:42:37.533190 24352 external_mini_cluster.cc:1428] Setting key 80a9a27170fe6aba7ef1d54985e9daf9
I20260806 20:42:37.539978 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.529863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:36939 (local address 127.23.200.62:42643)
0806 20:42:37.530082 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.530089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.531431 (+ 1342us) server_negotiation.cc:407] Connection header received
0806 20:42:37.533824 (+ 2393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.533832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.533923 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.534035 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:37.535333 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.536189 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.537656 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.538043 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.538135 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.539343 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:37.539348 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:37.539351 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:37.539367 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:37.539397 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:37.539408 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:37.539477 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.539666 (+ 189us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.539739 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 20:42:37.540853 10829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.530436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42643 (local address 127.23.200.3:36939)
0806 20:42:37.531278 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.531323 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.533410 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.534346 (+ 936us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.534359 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.534482 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:37.535158 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.535186 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.536324 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.536328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.537525 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.537537 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.538217 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.538222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.538480 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.539072 (+ 592us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:37.539074 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:37.539109 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:37.539181 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:37.539490 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:37.539497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:37.539502 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.539625 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.539880 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":251,"threads_started":1}
I20260806 20:42:37.542026 10827 heartbeater.cc:347] Connected to a master server at 127.23.200.62:42643
I20260806 20:42:37.542291 10827 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:37.543008 10827 heartbeater.cc:511] Master 127.23.200.62:42643 requested a full tablet report, sending...
I20260806 20:42:37.544322 10361 ts_manager.cc:194] Registered new tserver with Master: 4b616c3761a740b39f6f464d92279d0b (127.23.200.3:39515)
I20260806 20:42:37.545105 10361 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.3:36939
I20260806 20:42:37.548134 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:42:37.557571 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.552016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43558 (local address 127.23.200.62:42643)
0806 20:42:37.552158 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.552162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.552636 (+ 474us) server_negotiation.cc:407] Connection header received
0806 20:42:37.552754 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.552757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.552819 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.552895 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:37.553336 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.553970 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.555400 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.555813 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.555901 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.556848 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:37.556853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:37.556856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:37.556872 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:37.556900 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:37.556911 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:37.556976 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.557143 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.557229 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 20:42:37.560900 10361 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43558:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:42:37.567656 10361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:42:37.595604 10841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.579339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35111 (local address 127.0.0.1:35752)
0806 20:42:37.581540 (+ 2201us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.581557 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.581683 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.584123 (+ 2440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.584133 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.584173 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:37.584662 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.584678 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.588484 (+ 3806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.588493 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.592850 (+ 4357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.592865 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.594718 (+ 1853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.594724 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.594930 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.594977 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.595111 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.595284 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2074,"thread_start_us":2341,"threads_started":1}
I20260806 20:42:37.596596 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.579911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35752 (local address 127.23.200.2:35111)
0806 20:42:37.581168 (+ 1257us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.581178 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.582939 (+ 1761us) server_negotiation.cc:407] Connection header received
0806 20:42:37.583238 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.583250 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.583601 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.583808 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:37.584907 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.588228 (+ 3321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.593048 (+ 4820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.594531 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.594669 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.594902 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.596203 (+ 1301us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.596351 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1080,"thread_start_us":957,"threads_started":1}
I20260806 20:42:37.600596 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.579016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.23.200.1:33125)
0806 20:42:37.592642 (+ 13626us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.592653 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.592684 (+ 31us) server_negotiation.cc:407] Connection header received
0806 20:42:37.592793 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.592800 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.593021 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.593218 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:37.593790 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.595646 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.598621 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.599526 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.599676 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.599894 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.600074 (+ 180us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.600223 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13458,"thread_start_us":13342,"threads_started":1}
I20260806 20:42:37.600194 10623 tablet_service.cc:1467] Processing CreateTablet for tablet 60525187f2b64ea5b449495720a55b80 (DEFAULT_TABLE table=test-table [id=fa2f81a873e6401491de2b17affc8f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:37.601126 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.578919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59648)
0806 20:42:37.580280 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.580336 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.580509 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.593212 (+ 12703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.593217 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.593243 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:37.593592 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.593605 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.595803 (+ 2198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.595807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.598357 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.598366 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.599709 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.599716 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.599875 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.599907 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.600023 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.600144 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":111,"thread_start_us":896,"threads_started":1}
I20260806 20:42:37.601689 10843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.580717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:39515 (local address 127.0.0.1:52434)
0806 20:42:37.591802 (+ 11085us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.591824 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.591937 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.593571 (+ 1634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.593580 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.593611 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:37.593944 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.593952 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.595825 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.595828 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.598350 (+ 2522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.598366 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.600433 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.600439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.601030 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.601064 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.601160 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.601282 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10932,"mutex_wait_us":106,"thread_start_us":11314,"threads_started":1}
I20260806 20:42:37.601769 10623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60525187f2b64ea5b449495720a55b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:37.602451 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.581348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.23.200.3:39515)
0806 20:42:37.593022 (+ 11674us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.593030 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.593062 (+ 32us) server_negotiation.cc:407] Connection header received
0806 20:42:37.593152 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.593161 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.593363 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.593535 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:37.594124 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.595549 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.598568 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.599915 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.600042 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.600280 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.601901 (+ 1621us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.602067 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11508,"thread_start_us":11340,"threads_started":1}
I20260806 20:42:37.604617 10488 tablet_service.cc:1467] Processing CreateTablet for tablet 60525187f2b64ea5b449495720a55b80 (DEFAULT_TABLE table=test-table [id=fa2f81a873e6401491de2b17affc8f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:37.605644 10488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60525187f2b64ea5b449495720a55b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:37.605865 10759 tablet_service.cc:1467] Processing CreateTablet for tablet 60525187f2b64ea5b449495720a55b80 (DEFAULT_TABLE table=test-table [id=fa2f81a873e6401491de2b17affc8f88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:37.607236 10759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 60525187f2b64ea5b449495720a55b80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:37.622701 10846 tablet_bootstrap.cc:492] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Bootstrap starting.
I20260806 20:42:37.623497 10847 tablet_bootstrap.cc:492] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: Bootstrap starting.
I20260806 20:42:37.628585 10847 tablet_bootstrap.cc:654] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:37.628554 10848 tablet_bootstrap.cc:492] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Bootstrap starting.
I20260806 20:42:37.628978 10846 tablet_bootstrap.cc:654] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:37.629866 10847 log.cc:824] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:37.630250 10846 log.cc:824] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:37.632565 10847 tablet_bootstrap.cc:492] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: No bootstrap required, opened a new log
I20260806 20:42:37.632668 10848 tablet_bootstrap.cc:654] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:37.632853 10847 ts_tablet_manager.cc:1397] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260806 20:42:37.633350 10846 tablet_bootstrap.cc:492] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: No bootstrap required, opened a new log
I20260806 20:42:37.633613 10846 ts_tablet_manager.cc:1397] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260806 20:42:37.633805 10848 log.cc:824] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:37.636675 10848 tablet_bootstrap.cc:492] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: No bootstrap required, opened a new log
I20260806 20:42:37.636914 10848 ts_tablet_manager.cc:1397] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260806 20:42:37.640062 10555 heartbeater.cc:503] Master 127.23.200.62:42643 was elected leader, sending a full tablet report...
I20260806 20:42:37.640295 10847 raft_consensus.cc:348] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.640723 10847 raft_consensus.cc:374] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:37.640844 10847 raft_consensus.cc:729] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d1565943089418fb78f8b181ce137de, State: Initialized, Role: FOLLOWER
I20260806 20:42:37.641463 10847 consensus_queue.cc:260] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.644055 10846 raft_consensus.cc:348] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.644480 10846 raft_consensus.cc:374] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:37.644606 10846 raft_consensus.cc:729] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f6f85df031b4b1caf35015bdc464141, State: Initialized, Role: FOLLOWER
I20260806 20:42:37.645167 10846 consensus_queue.cc:260] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.647946 10848 raft_consensus.cc:348] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.648535 10848 raft_consensus.cc:374] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:37.648701 10848 raft_consensus.cc:729] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b616c3761a740b39f6f464d92279d0b, State: Initialized, Role: FOLLOWER
I20260806 20:42:37.649711 10848 consensus_queue.cc:260] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.650704 10847 ts_tablet_manager.cc:1428] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: Time spent starting tablet: real 0.018s user 0.008s sys 0.010s
I20260806 20:42:37.652856 10691 heartbeater.cc:503] Master 127.23.200.62:42643 was elected leader, sending a full tablet report...
I20260806 20:42:37.653627 10846 ts_tablet_manager.cc:1428] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Time spent starting tablet: real 0.020s user 0.017s sys 0.003s
I20260806 20:42:37.654611 10827 heartbeater.cc:503] Master 127.23.200.62:42643 was elected leader, sending a full tablet report...
I20260806 20:42:37.655138 10848 ts_tablet_manager.cc:1428] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Time spent starting tablet: real 0.018s user 0.016s sys 0.002s
I20260806 20:42:37.691876 10852 raft_consensus.cc:482] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:37.692269 10852 raft_consensus.cc:504] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.693900 10852 leader_election.cc:288] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b616c3761a740b39f6f464d92279d0b (127.23.200.3:39515), 9f6f85df031b4b1caf35015bdc464141 (127.23.200.2:35111)
I20260806 20:42:37.703161 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.694483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:40917 (local address 127.23.200.2:35111)
0806 20:42:37.694691 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.694700 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.695426 (+ 726us) server_negotiation.cc:407] Connection header received
0806 20:42:37.695569 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.695575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.695655 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.695736 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:37.696425 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.697541 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.700891 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.702108 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.702270 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.702369 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.702776 (+ 407us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.702903 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 20:42:37.703317 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.694259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:35111 (local address 127.23.200.1:40917)
0806 20:42:37.695318 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.695334 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.695446 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.695761 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.695771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.695811 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:37.696259 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.696266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.697675 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.697678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.700666 (+ 2988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.700681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.702413 (+ 1732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.702420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.702572 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.702623 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.702719 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.702896 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":958,"mutex_wait_us":375,"spinlock_wait_cycles":26240,"thread_start_us":390,"threads_started":1}
I20260806 20:42:37.703701 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.694443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:58835 (local address 127.23.200.3:39515)
0806 20:42:37.694860 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.694865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.695196 (+ 331us) server_negotiation.cc:407] Connection header received
0806 20:42:37.695444 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.695452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.695553 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.695646 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:37.696399 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.697541 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.700911 (+ 3370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.702134 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.702273 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.702368 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.703371 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.703503 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279}
I20260806 20:42:37.703745 10856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.694224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:39515 (local address 127.23.200.1:58835)
0806 20:42:37.695053 (+ 829us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.695080 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:37.695272 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:37.695668 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:37.695672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:37.695691 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:37.696226 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.696244 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.697673 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.697678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.700670 (+ 2992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:37.700681 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.702377 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.702391 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:37.702568 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.702631 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 20:42:37.702719 (+ 88us) client_negotiation.cc:240] Negotiation successful
0806 20:42:37.702896 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":295,"spinlock_wait_cycles":768,"thread_start_us":806,"threads_started":1}
I20260806 20:42:37.704523 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60525187f2b64ea5b449495720a55b80" candidate_uuid: "1d1565943089418fb78f8b181ce137de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6f85df031b4b1caf35015bdc464141" is_pre_election: true
I20260806 20:42:37.705134 10643 raft_consensus.cc:2454] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d1565943089418fb78f8b181ce137de in term 0.
I20260806 20:42:37.704984 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60525187f2b64ea5b449495720a55b80" candidate_uuid: "1d1565943089418fb78f8b181ce137de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b616c3761a740b39f6f464d92279d0b" is_pre_election: true
I20260806 20:42:37.705542 10780 raft_consensus.cc:2454] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1d1565943089418fb78f8b181ce137de in term 0.
I20260806 20:42:37.705832 10443 leader_election.cc:302] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1565943089418fb78f8b181ce137de, 9f6f85df031b4b1caf35015bdc464141; no voters:
I20260806 20:42:37.706269 10852 raft_consensus.cc:2781] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:42:37.706460 10852 raft_consensus.cc:482] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:42:37.706584 10852 raft_consensus.cc:3037] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:37.708760 10852 raft_consensus.cc:504] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.709419 10852 leader_election.cc:288] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [CANDIDATE]: Term 1 election: Requested vote from peers 4b616c3761a740b39f6f464d92279d0b (127.23.200.3:39515), 9f6f85df031b4b1caf35015bdc464141 (127.23.200.2:35111)
I20260806 20:42:37.709766 10780 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60525187f2b64ea5b449495720a55b80" candidate_uuid: "1d1565943089418fb78f8b181ce137de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b616c3761a740b39f6f464d92279d0b"
I20260806 20:42:37.709985 10780 raft_consensus.cc:3037] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:37.710064 10643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "60525187f2b64ea5b449495720a55b80" candidate_uuid: "1d1565943089418fb78f8b181ce137de" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f6f85df031b4b1caf35015bdc464141"
I20260806 20:42:37.710283 10643 raft_consensus.cc:3037] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:37.712296 10780 raft_consensus.cc:2454] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d1565943089418fb78f8b181ce137de in term 1.
I20260806 20:42:37.712731 10643 raft_consensus.cc:2454] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1d1565943089418fb78f8b181ce137de in term 1.
I20260806 20:42:37.712724 10440 leader_election.cc:302] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d1565943089418fb78f8b181ce137de, 4b616c3761a740b39f6f464d92279d0b; no voters:
I20260806 20:42:37.713054 10852 raft_consensus.cc:2781] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:37.713486 10852 raft_consensus.cc:686] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 1 LEADER]: Becoming Leader. State: Replica: 1d1565943089418fb78f8b181ce137de, State: Running, Role: LEADER
I20260806 20:42:37.713992 10852 consensus_queue.cc:237] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } }
I20260806 20:42:37.718079 10360 catalog_manager.cc:5495] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de reported cstate change: term changed from 0 to 1, leader changed from <none> to 1d1565943089418fb78f8b181ce137de (127.23.200.1). New cstate: current_term: 1 leader_uuid: "1d1565943089418fb78f8b181ce137de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d1565943089418fb78f8b181ce137de" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 33125 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 } health_report { overall_health: UNKNOWN } } }
I20260806 20:42:37.746729 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.738963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.23.200.1:33125)
0806 20:42:37.739166 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.739174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.739192 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:42:37.739435 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.739439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.739558 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.739713 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:42:37.740217 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.741467 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.743937 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.744649 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.744800 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.745429 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:42:37.746106 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:42:37.746277 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.746374 (+ 97us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.746452 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:42:37.755255 10643 raft_consensus.cc:1261] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 1 FOLLOWER]: Refusing update from remote peer 1d1565943089418fb78f8b181ce137de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:37.755264 10780 raft_consensus.cc:1261] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 1 FOLLOWER]: Refusing update from remote peer 1d1565943089418fb78f8b181ce137de: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:37.756191 10857 consensus_queue.cc:1035] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b616c3761a740b39f6f464d92279d0b" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:37.756537 10852 consensus_queue.cc:1035] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f6f85df031b4b1caf35015bdc464141" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 35111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:37.768649 10860 mvcc.cc:205] Tried to move back new op lower bound from 7315656530955681792 to 7315656530800693248. Current Snapshot: MvccSnapshot[applied={T|T < 7315656530955681792}]
I20260806 20:42:37.773145 10861 mvcc.cc:205] Tried to move back new op lower bound from 7315656530955681792 to 7315656530800693248. Current Snapshot: MvccSnapshot[applied={T|T < 7315656530955681792}]
I20260806 20:42:37.784543 10360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:37.784811 10360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43558:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:37.788234 10360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6e40d2dc6fb14846a28d72b1b8f5825d: Sending DeleteTablet for 3 replicas of tablet 60525187f2b64ea5b449495720a55b80
I20260806 20:42:37.789448 10623 tablet_service.cc:1514] Processing DeleteTablet for tablet 60525187f2b64ea5b449495720a55b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:37 UTC) from {username='slave'} at 127.0.0.1:35752
I20260806 20:42:37.789474 10759 tablet_service.cc:1514] Processing DeleteTablet for tablet 60525187f2b64ea5b449495720a55b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:37 UTC) from {username='slave'} at 127.0.0.1:52434
I20260806 20:42:37.789480 10488 tablet_service.cc:1514] Processing DeleteTablet for tablet 60525187f2b64ea5b449495720a55b80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:37 UTC) from {username='slave'} at 127.0.0.1:59648
I20260806 20:42:37.791633 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10423
I20260806 20:42:37.802206 10870 tablet_replica.cc:330] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: stopping tablet replica
I20260806 20:42:37.802695 10870 raft_consensus.cc:2229] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:37.803231 10870 raft_consensus.cc:2258] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:37.803975 10869 tablet_replica.cc:330] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: stopping tablet replica
I20260806 20:42:37.804359 10869 raft_consensus.cc:2229] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:37.804811 10869 raft_consensus.cc:2258] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:37.805773 10871 tablet_replica.cc:330] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: stopping tablet replica
I20260806 20:42:37.806154 10871 raft_consensus.cc:2229] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:42:37.806838 10871 raft_consensus.cc:2258] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:37.807309 10870 ts_tablet_manager.cc:1905] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:37.808914 10869 ts_tablet_manager.cc:1905] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:37.811605 10840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.793189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.23.200.1:33125)
0806 20:42:37.793411 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.793420 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.802470 (+ 9050us) server_negotiation.cc:407] Connection header received
0806 20:42:37.802809 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.802820 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.802962 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.803059 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:37.803899 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.805433 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.807575 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.808253 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.808351 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.809850 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:37.809861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:37.809883 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:37.810643 (+ 760us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:37.810696 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:37.810722 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:37.810815 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.811098 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.811208 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:42:37.812134 10871 ts_tablet_manager.cc:1905] T 60525187f2b64ea5b449495720a55b80 P 1d1565943089418fb78f8b181ce137de: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:37.813052 10550 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:37.815645 10870 ts_tablet_manager.cc:1918] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:37.815814 10870 log.cc:1196] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/60525187f2b64ea5b449495720a55b80
I20260806 20:42:37.816627 10870 ts_tablet_manager.cc:1939] T 60525187f2b64ea5b449495720a55b80 P 4b616c3761a740b39f6f464d92279d0b: Deleting consensus metadata
I20260806 20:42:37.818354 10345 catalog_manager.cc:4841] TS 4b616c3761a740b39f6f464d92279d0b (127.23.200.3:39515): tablet 60525187f2b64ea5b449495720a55b80 (table test-table [id=fa2f81a873e6401491de2b17affc8f88]) successfully deleted
I20260806 20:42:37.819413 10869 ts_tablet_manager.cc:1918] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:37.819582 10869 log.cc:1196] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/60525187f2b64ea5b449495720a55b80
I20260806 20:42:37.820138 10869 ts_tablet_manager.cc:1939] T 60525187f2b64ea5b449495720a55b80 P 9f6f85df031b4b1caf35015bdc464141: Deleting consensus metadata
I20260806 20:42:37.821760 10348 catalog_manager.cc:4841] TS 9f6f85df031b4b1caf35015bdc464141 (127.23.200.2:35111): tablet 60525187f2b64ea5b449495720a55b80 (table test-table [id=fa2f81a873e6401491de2b17affc8f88]) successfully deleted
I20260806 20:42:37.909538 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10423
I20260806 20:42:37.928467 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10559
W20260806 20:42:37.928498 10345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:33125 (error 108)
I20260806 20:42:37.930423 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.929938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59664)
0806 20:42:37.930104 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.930224 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 20:42:37.930656 10345 catalog_manager.cc:4584] TS 1d1565943089418fb78f8b181ce137de (127.23.200.1:33125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 60525187f2b64ea5b449495720a55b80: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
I20260806 20:42:37.935659 10842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.929321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35762 (local address 127.23.200.2:35111)
0806 20:42:37.929502 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:37.929521 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:37.929541 (+ 20us) server_negotiation.cc:407] Connection header received
0806 20:42:37.929646 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:37.929651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:37.929751 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:37.929871 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:37.930328 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.931300 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.932595 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:37.933002 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:37.933143 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:37.934152 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:37.934161 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:37.934174 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:37.934901 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:37.934947 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:37.934966 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:37.935097 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:37.935230 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 20:42:37.935334 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 20:42:37.936373 10686 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:37.984131 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.983682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59668)
0806 20:42:37.983888 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.983978 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 20:42:37.985105 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:37.984760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59672)
0806 20:42:37.984907 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:37.984980 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 20:42:38.025684 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10559
I20260806 20:42:38.044868 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10694
I20260806 20:42:38.045693 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:38.045157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59682)
0806 20:42:38.045361 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:38.045477 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 20:42:38.046742 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:38.046414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59692)
0806 20:42:38.046555 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:38.046630 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260806 20:42:38.052033 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:38.045664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.23.200.3:39515)
0806 20:42:38.045848 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:38.045854 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:38.045876 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:42:38.046115 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:38.046121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:38.046218 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:38.046350 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:38.046770 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:38.047570 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:38.048868 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:38.049272 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:38.049412 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:38.050698 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:38.050707 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:38.050723 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:38.051326 (+ 603us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:38.051371 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:38.051391 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:38.051512 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:38.051647 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 20:42:38.051748 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260806 20:42:38.052635 10822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:38.127545 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:38.126962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59698)
0806 20:42:38.127211 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:38.127329 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260806 20:42:38.128660 10839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:38.128228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:33125 (local address 127.0.0.1:59702)
0806 20:42:38.128444 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:38.128527 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:33125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260806 20:42:38.141141 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10694
I20260806 20:42:38.159032 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10329
I20260806 20:42:38.160470 10392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:38.271236 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10329
2026-08-06T20:42:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2750 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:38 dist-test-slave-tm1g krb5kdc[10880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:38 dist-test-slave-tm1g krb5kdc[10880](info): set up 2 sockets
Aug 06 20:42:38 dist-test-slave-tm1g krb5kdc[10880](info): commencing operation
krb5kdc: starting...
W20260806 20:42:39.389343 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:42:39 dist-test-slave-tm1g krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:42:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:42:39.633163 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36069
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:36069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:39.964596 10894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:39.964912 10894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:39.965080 10894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:39.972846 10894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:39.972975 10894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:39.973071 10894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:39.973182 10894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:39.973263 10894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:39.982949 10894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:36069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:39.984239 10894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:39.985544 10894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:39.997227 10900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:39.997195 10899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:39.997431 10902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:39.997892 10894 server_base.cc:1034] running on GCE node
I20260806 20:42:39.998713 10894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:40.000032 10894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:40.001273 10894 hybrid_clock.cc:648] HybridClock initialized: now 1786048960001205 us; error 38 us; skew 500 ppm
I20260806 20:42:40.001637 10894 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:40 dist-test-slave-tm1g krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048960, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:40.004739 10894 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:42:40.006119 10894 webserver.cc:460] Webserver started at http://127.23.200.62:34815/ using document root <none> and password file <none>
I20260806 20:42:40.006842 10894 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:40.006975 10894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:40.007294 10894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:40.010149 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "bbe26cf098a241f597cc50234f79df29"
format_stamp: "Formatted at 2026-08-06 20:42:40 on dist-test-slave-tm1g"
server_key: "1a1f812c1ed5334c9901ecd98b6ce2bd"
server_key_iv: "cc4a15b78d85f576a822e635d4416933"
server_key_version: "encryptionkey@0"
I20260806 20:42:40.010963 10894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "bbe26cf098a241f597cc50234f79df29"
format_stamp: "Formatted at 2026-08-06 20:42:40 on dist-test-slave-tm1g"
server_key: "1a1f812c1ed5334c9901ecd98b6ce2bd"
server_key_iv: "cc4a15b78d85f576a822e635d4416933"
server_key_version: "encryptionkey@0"
I20260806 20:42:40.015861 10894 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:42:40.019145 10909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:40.020357 10894 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260806 20:42:40.020535 10894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "bbe26cf098a241f597cc50234f79df29"
format_stamp: "Formatted at 2026-08-06 20:42:40 on dist-test-slave-tm1g"
server_key: "1a1f812c1ed5334c9901ecd98b6ce2bd"
server_key_iv: "cc4a15b78d85f576a822e635d4416933"
server_key_version: "encryptionkey@0"
I20260806 20:42:40.020730 10894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:40.043087 10894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:40.046993 10894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:40.047278 10894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:40.065769 10894 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:36069
I20260806 20:42:40.065802 10961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:36069 every 8 connection(s)
I20260806 20:42:40.067077 10894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:40.070681 10962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:40.073782 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10894
I20260806 20:42:40.073984 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:40.074486 24352 external_mini_cluster.cc:1428] Setting key 3035ab0634ff1966b32bc6f3a146c897
I20260806 20:42:40.082796 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29: Bootstrap starting.
I20260806 20:42:40.086457 10962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:40.087747 10962 log.cc:824] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29: Log is configured to *not* fsync() on all Append() calls
Aug 06 20:42:40 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:40.090507 10962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29: No bootstrap required, opened a new log
I20260806 20:42:40.097301 10965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:40.076285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59902 (local address 127.23.200.62:36069)
0806 20:42:40.077045 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:40.077056 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:40.077093 (+ 37us) server_negotiation.cc:407] Connection header received
0806 20:42:40.078837 (+ 1744us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:40.078858 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:40.079228 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:40.079537 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:40.081372 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:40.082731 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:40.084514 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:40.085129 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:40.085257 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:40.090135 (+ 4878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:40.090142 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:40.090155 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:40.090192 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:40.093295 (+ 3103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:40.094018 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:40.094027 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:40.094038 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:40.094191 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:40.094609 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:40.094613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:40.094618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:40.095145 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:40.095475 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:40.095934 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 20:42:40.096340 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":390,"threads_started":1}
I20260806 20:42:40.098631 10962 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe26cf098a241f597cc50234f79df29" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36069 } }
I20260806 20:42:40.099012 10962 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:40.099133 10962 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbe26cf098a241f597cc50234f79df29, State: Initialized, Role: FOLLOWER
I20260806 20:42:40.099666 10962 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe26cf098a241f597cc50234f79df29" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36069 } }
I20260806 20:42:40.099936 10962 raft_consensus.cc:388] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:40.100100 10962 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:40.100297 10962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:40.102259 10962 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe26cf098a241f597cc50234f79df29" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36069 } }
I20260806 20:42:40.102752 10962 leader_election.cc:302] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bbe26cf098a241f597cc50234f79df29; no voters:
I20260806 20:42:40.103262 10962 leader_election.cc:288] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:40.103425 10967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:40.104144 10967 raft_consensus.cc:686] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [term 1 LEADER]: Becoming Leader. State: Replica: bbe26cf098a241f597cc50234f79df29, State: Running, Role: LEADER
I20260806 20:42:40.104652 10967 consensus_queue.cc:237] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe26cf098a241f597cc50234f79df29" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36069 } }
I20260806 20:42:40.105239 10962 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:40.107885 10968 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bbe26cf098a241f597cc50234f79df29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe26cf098a241f597cc50234f79df29" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36069 } } }
I20260806 20:42:40.108285 10968 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:40.108915 10969 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bbe26cf098a241f597cc50234f79df29. Latest consensus state: current_term: 1 leader_uuid: "bbe26cf098a241f597cc50234f79df29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bbe26cf098a241f597cc50234f79df29" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36069 } } }
I20260806 20:42:40.109237 10969 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:40.111268 10974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:40.117578 10974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:40.126504 10974 catalog_manager.cc:1347] Generated new cluster ID: 6a3e6dfa90984a0f9ff916c5225e7de7
I20260806 20:42:40.126628 10974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:40.146628 10974 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:40.147945 10974 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:40.163789 10974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29: Generated new TSK 0
I20260806 20:42:40.164630 10974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:42:40.231216 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36069
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:40.568838 10988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:40.569093 10988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:40.569188 10988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:40.576172 10988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:40.576303 10988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:40.576459 10988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:40.585569 10988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:40.586755 10988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:40.587949 10988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:40.600880 10993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:40.600944 10994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:40.600953 10996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:40.602061 10988 server_base.cc:1034] running on GCE node
I20260806 20:42:40.602684 10988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:40.603550 10988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:40.604776 10988 hybrid_clock.cc:648] HybridClock initialized: now 1786048960604736 us; error 50 us; skew 500 ppm
I20260806 20:42:40.605075 10988 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:40 dist-test-slave-tm1g krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048960, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:40.608451 10988 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:42:40.609915 10988 webserver.cc:460] Webserver started at http://127.23.200.1:36067/ using document root <none> and password file <none>
I20260806 20:42:40.610669 10988 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:40.610827 10988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:40.611183 10988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:40.614058 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "8e51055df69e403fb1ad58f903bcccd5"
format_stamp: "Formatted at 2026-08-06 20:42:40 on dist-test-slave-tm1g"
server_key: "6aa4d7c0b7ff04f109752d006ee2a959"
server_key_iv: "cbcaa792d268f33498cbd6f4d5a134e3"
server_key_version: "encryptionkey@0"
I20260806 20:42:40.614874 10988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "8e51055df69e403fb1ad58f903bcccd5"
format_stamp: "Formatted at 2026-08-06 20:42:40 on dist-test-slave-tm1g"
server_key: "6aa4d7c0b7ff04f109752d006ee2a959"
server_key_iv: "cbcaa792d268f33498cbd6f4d5a134e3"
server_key_version: "encryptionkey@0"
I20260806 20:42:40.619709 10988 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:42:40.623085 11003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:40.624400 10988 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:42:40.624571 10988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "8e51055df69e403fb1ad58f903bcccd5"
format_stamp: "Formatted at 2026-08-06 20:42:40 on dist-test-slave-tm1g"
server_key: "6aa4d7c0b7ff04f109752d006ee2a959"
server_key_iv: "cbcaa792d268f33498cbd6f4d5a134e3"
server_key_version: "encryptionkey@0"
I20260806 20:42:40.624765 10988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:40.637596 10988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:40.641793 10988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:40.642117 10988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:40.643034 10988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:40.644263 10988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:40.644368 10988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:40.644519 10988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:40.644598 10988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:40.677350 10988 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:39413
I20260806 20:42:40.677399 11115 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:39413 every 8 connection(s)
I20260806 20:42:40.678700 10988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:40.688123 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 10988
I20260806 20:42:40.688306 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:42:40.688771 24352 external_mini_cluster.cc:1428] Setting key 408efdea9dd52edb235f072a44c88373
Aug 06 20:42:40 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048960, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:40.699352 11118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:40.681254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:42657 (local address 127.23.200.62:36069)
0806 20:42:40.681685 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:40.681689 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:40.682713 (+ 1024us) server_negotiation.cc:407] Connection header received
0806 20:42:40.684506 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:40.684512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:40.684598 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:40.684733 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:40.686201 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:40.687023 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:40.688685 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:40.689181 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:40.689320 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:40.693439 (+ 4119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:40.693444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:40.693448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:40.693479 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:40.695972 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:40.696928 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:40.696932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:40.696934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:40.697056 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:40.697530 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:40.697533 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:40.697535 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:40.697785 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:40.697895 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:40.698800 (+ 905us) server_negotiation.cc:299] Negotiation successful
0806 20:42:40.699075 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260806 20:42:40.700760 11119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:40.681786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36069 (local address 127.23.200.1:42657)
0806 20:42:40.682556 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:40.682595 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:40.684243 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:40.684961 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:40.684969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:40.685475 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:40.686027 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:40.686046 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:40.687202 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:40.687212 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:40.688548 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:40.688561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:40.689420 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:40.689429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:40.689708 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:40.690529 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:40.690552 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:40.693236 (+ 2684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:40.696132 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:40.696143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:40.696161 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:40.696780 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:40.697165 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:40.697173 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:40.697180 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:40.697425 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:40.697909 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:40.697920 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:40.698400 (+ 480us) client_negotiation.cc:769] Sending connection context
0806 20:42:40.698720 (+ 320us) client_negotiation.cc:240] Negotiation successful
0806 20:42:40.699298 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":280,"threads_started":1}
I20260806 20:42:40.702486 11116 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36069
I20260806 20:42:40.702807 11116 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:40.703528 11116 heartbeater.cc:511] Master 127.23.200.62:36069 requested a full tablet report, sending...
I20260806 20:42:40.705760 10926 ts_manager.cc:194] Registered new tserver with Master: 8e51055df69e403fb1ad58f903bcccd5 (127.23.200.1:39413)
I20260806 20:42:40.708032 10926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:42657
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:42:40.750437 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36069
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:41.095193 11122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:41.095453 11122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:41.095610 11122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:41.105237 11122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:41.105365 11122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:41.105574 11122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:41.115545 11122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:41.116827 11122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:41.118145 11122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:41.131639 11129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:41.131667 11131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:41.131661 11128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:41.132669 11122 server_base.cc:1034] running on GCE node
I20260806 20:42:41.133201 11122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:41.134037 11122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:41.135277 11122 hybrid_clock.cc:648] HybridClock initialized: now 1786048961135227 us; error 49 us; skew 500 ppm
I20260806 20:42:41.135697 11122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:41 dist-test-slave-tm1g krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048961, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:41.139019 11122 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:42:41.140342 11122 webserver.cc:460] Webserver started at http://127.23.200.2:41065/ using document root <none> and password file <none>
I20260806 20:42:41.140959 11122 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:41.141060 11122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:41.141312 11122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:41.144045 11122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "ec3ff77e061d44a3b6154adfc0ce13aa"
format_stamp: "Formatted at 2026-08-06 20:42:41 on dist-test-slave-tm1g"
server_key: "a6e966856ac8a14745b928310a0716df"
server_key_iv: "d2272d12bb5bc25c0756f986bc6046bb"
server_key_version: "encryptionkey@0"
I20260806 20:42:41.144757 11122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "ec3ff77e061d44a3b6154adfc0ce13aa"
format_stamp: "Formatted at 2026-08-06 20:42:41 on dist-test-slave-tm1g"
server_key: "a6e966856ac8a14745b928310a0716df"
server_key_iv: "d2272d12bb5bc25c0756f986bc6046bb"
server_key_version: "encryptionkey@0"
I20260806 20:42:41.149537 11122 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260806 20:42:41.152769 11138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:41.153970 11122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 20:42:41.154121 11122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "ec3ff77e061d44a3b6154adfc0ce13aa"
format_stamp: "Formatted at 2026-08-06 20:42:41 on dist-test-slave-tm1g"
server_key: "a6e966856ac8a14745b928310a0716df"
server_key_iv: "d2272d12bb5bc25c0756f986bc6046bb"
server_key_version: "encryptionkey@0"
I20260806 20:42:41.154301 11122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:41.175462 11122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:41.179642 11122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:41.179939 11122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:41.180853 11122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:41.182112 11122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:41.182186 11122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:41.182377 11122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:41.182432 11122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 20:42:41.215579 11122 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:43585
I20260806 20:42:41.215682 11251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:43585 every 8 connection(s)
I20260806 20:42:41.216960 11122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:41.226759 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 11122
I20260806 20:42:41.226934 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:42:41.227442 24352 external_mini_cluster.cc:1428] Setting key 8cc34caf40e28b6d6f93021b202d3cf5
Aug 06 20:42:41 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048961, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:41.236670 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.219485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:56327 (local address 127.23.200.62:36069)
0806 20:42:41.219908 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.219913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.220969 (+ 1056us) server_negotiation.cc:407] Connection header received
0806 20:42:41.222751 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.222756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.222825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.222920 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:41.224329 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.225132 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.226731 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.227324 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.227461 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.231091 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:41.231097 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:41.231104 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:41.231142 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:41.233646 (+ 2504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:41.234395 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:41.234401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:41.234405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:41.234558 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:41.234991 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:41.234996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:41.235001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:41.235265 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:41.235426 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.236083 (+ 657us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.236296 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":210,"threads_started":1}
I20260806 20:42:41.237568 11255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.220074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36069 (local address 127.23.200.2:56327)
0806 20:42:41.220809 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.220850 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.222517 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.223118 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.223125 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.223599 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:41.224130 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.224146 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.225315 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.225319 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.226591 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.226603 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.227546 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.227553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.227785 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.228498 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:41.228522 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:41.230878 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:41.233798 (+ 2920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:41.233806 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:41.233820 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:41.234250 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:41.234670 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:41.234675 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:41.234678 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:41.234854 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:41.235398 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:41.235406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:41.235732 (+ 326us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.235978 (+ 246us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.236465 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":296,"threads_started":1}
I20260806 20:42:41.239006 11252 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36069
I20260806 20:42:41.239298 11252 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:41.240080 11252 heartbeater.cc:511] Master 127.23.200.62:36069 requested a full tablet report, sending...
I20260806 20:42:41.241576 10926 ts_manager.cc:194] Registered new tserver with Master: ec3ff77e061d44a3b6154adfc0ce13aa (127.23.200.2:43585)
I20260806 20:42:41.242430 10926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:56327
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:42:41.281410 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36069
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:41.604447 11258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:41.604684 11258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:41.604774 11258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:41.612112 11258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:41.612206 11258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:41.612313 11258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:41.621938 11258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:41.623171 11258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:41.624440 11258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:41.637305 11267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:41.637422 11263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:41.637444 11264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:41.638271 11258 server_base.cc:1034] running on GCE node
I20260806 20:42:41.638903 11258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:41.639746 11258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:41.641024 11258 hybrid_clock.cc:648] HybridClock initialized: now 1786048961640979 us; error 51 us; skew 500 ppm
I20260806 20:42:41.641355 11258 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:41 dist-test-slave-tm1g krb5kdc[10880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048961, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:41.644657 11258 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:42:41.647039 11258 webserver.cc:460] Webserver started at http://127.23.200.3:43371/ using document root <none> and password file <none>
I20260806 20:42:41.647734 11258 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:41.647838 11258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:41.648167 11258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:41.651022 11258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "47c8c7396bc7413ba5916aa0f39fe01f"
format_stamp: "Formatted at 2026-08-06 20:42:41 on dist-test-slave-tm1g"
server_key: "b4db43f2005abb95bd5084056a3e2254"
server_key_iv: "340dfbf7e8ff3f170cf2d0001f4672e5"
server_key_version: "encryptionkey@0"
I20260806 20:42:41.651769 11258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "47c8c7396bc7413ba5916aa0f39fe01f"
format_stamp: "Formatted at 2026-08-06 20:42:41 on dist-test-slave-tm1g"
server_key: "b4db43f2005abb95bd5084056a3e2254"
server_key_iv: "340dfbf7e8ff3f170cf2d0001f4672e5"
server_key_version: "encryptionkey@0"
I20260806 20:42:41.656824 11258 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:42:41.660288 11274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:41.661576 11258 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260806 20:42:41.661765 11258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "47c8c7396bc7413ba5916aa0f39fe01f"
format_stamp: "Formatted at 2026-08-06 20:42:41 on dist-test-slave-tm1g"
server_key: "b4db43f2005abb95bd5084056a3e2254"
server_key_iv: "340dfbf7e8ff3f170cf2d0001f4672e5"
server_key_version: "encryptionkey@0"
I20260806 20:42:41.661980 11258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:41.679021 11258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:41.682991 11258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:41.683272 11258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:41.684161 11258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:41.685422 11258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:41.685534 11258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:41.685684 11258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:41.685757 11258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:41.710978 11116 heartbeater.cc:503] Master 127.23.200.62:36069 was elected leader, sending a full tablet report...
I20260806 20:42:41.719869 11258 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:46815
I20260806 20:42:41.719914 11389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:46815 every 8 connection(s)
I20260806 20:42:41.721117 11258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:41.723582 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 11258
I20260806 20:42:41.723789 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:42:41.724344 24352 external_mini_cluster.cc:1428] Setting key 9ef169d82a7091bf977aae2f4014087e
Aug 06 20:42:41 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048961, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:41.741771 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.723625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:53821 (local address 127.23.200.62:36069)
0806 20:42:41.723852 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.723859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.725448 (+ 1589us) server_negotiation.cc:407] Connection header received
0806 20:42:41.727731 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.727735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.727800 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.727882 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:41.729463 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.730241 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.732070 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.732484 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.732582 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.735798 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:41.735803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:41.735806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:41.735836 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:41.738576 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:41.739329 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:41.739333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:41.739335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:41.739500 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:41.740062 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:41.740067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:41.740074 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:41.740431 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:41.740596 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.741249 (+ 653us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.741463 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 20:42:41.742835 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.724328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36069 (local address 127.23.200.3:53821)
0806 20:42:41.725255 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.725330 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.727504 (+ 2174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.728135 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.728142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.728701 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:41.729293 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.729312 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.730452 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.730456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.731923 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.731935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.732669 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.732674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.732849 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.733515 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:41.733540 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:41.735615 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:41.738713 (+ 3098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:41.738720 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:41.738733 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:41.739209 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:41.739613 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:41.739618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:41.739640 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:41.739919 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:41.740567 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:41.740576 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:41.740884 (+ 308us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.741138 (+ 254us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.741610 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":372,"threads_started":1}
I20260806 20:42:41.744560 11390 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36069
I20260806 20:42:41.744918 11390 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:41.745832 11390 heartbeater.cc:511] Master 127.23.200.62:36069 requested a full tablet report, sending...
I20260806 20:42:41.747545 10926 ts_manager.cc:194] Registered new tserver with Master: 47c8c7396bc7413ba5916aa0f39fe01f (127.23.200.3:46815)
I20260806 20:42:41.748708 10926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:53821
I20260806 20:42:41.752296 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:42:41.769122 11254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.756493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.23.200.62:36069)
0806 20:42:41.756670 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.756678 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.757138 (+ 460us) server_negotiation.cc:407] Connection header received
0806 20:42:41.757272 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.757275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.757350 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.757429 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:41.758895 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.759601 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.760868 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.761251 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.761359 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.763927 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:41.763932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:41.763935 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:41.763982 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:41.766518 (+ 2536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:41.767277 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:41.767282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:41.767284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:41.767446 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:41.767889 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:41.767893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:41.767895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:41.768182 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:41.768423 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.768721 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.768857 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 20:42:41.772956 10926 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:42:41.779681 10926 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:42:41.802265 11402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.791871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:43585 (local address 127.0.0.1:60322)
0806 20:42:41.792859 (+ 988us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.792907 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.793117 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.794416 (+ 1299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.794424 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.794463 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:41.794909 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.794925 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.797248 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.797253 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.799367 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.799381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.801275 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.801282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.801533 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.801615 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.801718 (+ 103us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.801885 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":457,"threads_started":1}
I20260806 20:42:41.803329 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.792044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.23.200.2:43585)
0806 20:42:41.793753 (+ 1709us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.793763 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.793794 (+ 31us) server_negotiation.cc:407] Connection header received
0806 20:42:41.793912 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.793919 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.794172 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.794390 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:41.795481 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.797105 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.799534 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.801026 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.802803 (+ 1777us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.802951 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.803004 (+ 53us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.803114 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1537,"thread_start_us":1418,"threads_started":1}
I20260806 20:42:41.804705 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.792538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56488)
0806 20:42:41.794028 (+ 1490us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.794054 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.794162 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.796262 (+ 2100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.796271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.796310 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:41.797553 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.797569 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.799298 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.799304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.801288 (+ 1984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.801308 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.803962 (+ 2654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.803967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.804077 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.804100 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.804205 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.804325 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1373,"mutex_wait_us":60,"thread_start_us":1245,"threads_started":1}
I20260806 20:42:41.804720 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.793022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.23.200.1:39413)
0806 20:42:41.795671 (+ 2649us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.795682 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.795710 (+ 28us) server_negotiation.cc:407] Connection header received
0806 20:42:41.795821 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.795831 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.796045 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.796230 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:41.797762 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.799145 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.801513 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.803478 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.803614 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.803835 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.804251 (+ 416us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.804414 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2478,"thread_start_us":2315,"threads_started":1}
I20260806 20:42:41.806900 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.793106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:46815 (local address 127.0.0.1:55634)
0806 20:42:41.796779 (+ 3673us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.796797 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.796928 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.797833 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.797838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.797863 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:41.798244 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.798256 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.800101 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.800106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.803714 (+ 3608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.803729 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.806022 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.806030 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.806212 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.806247 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.806450 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.806583 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3562,"thread_start_us":3435,"threads_started":1}
I20260806 20:42:41.807230 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.793696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55634 (local address 127.23.200.3:46815)
0806 20:42:41.795140 (+ 1444us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.795151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.797043 (+ 1892us) server_negotiation.cc:407] Connection header received
0806 20:42:41.797144 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.797154 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.797372 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.797558 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:41.798436 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.799947 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.803853 (+ 3906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.805808 (+ 1955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.805949 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.806164 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.806510 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.806690 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1258,"thread_start_us":1094,"threads_started":1}
I20260806 20:42:41.807404 11186 tablet_service.cc:1467] Processing CreateTablet for tablet bf0634d3f6c042359908328ace446657 (DEFAULT_TABLE table=test-table [id=53763c141b874689a30b0568fba71cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:41.808866 11186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0634d3f6c042359908328ace446657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:41.808589 11051 tablet_service.cc:1467] Processing CreateTablet for tablet bf0634d3f6c042359908328ace446657 (DEFAULT_TABLE table=test-table [id=53763c141b874689a30b0568fba71cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:41.809619 11051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0634d3f6c042359908328ace446657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:41.810760 11323 tablet_service.cc:1467] Processing CreateTablet for tablet bf0634d3f6c042359908328ace446657 (DEFAULT_TABLE table=test-table [id=53763c141b874689a30b0568fba71cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:41.812265 11323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf0634d3f6c042359908328ace446657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:41.820804 11408 tablet_bootstrap.cc:492] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Bootstrap starting.
I20260806 20:42:41.823329 11409 tablet_bootstrap.cc:492] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Bootstrap starting.
I20260806 20:42:41.824501 11408 tablet_bootstrap.cc:654] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:41.825810 11408 log.cc:824] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:41.826920 11409 tablet_bootstrap.cc:654] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:41.827350 11411 tablet_bootstrap.cc:492] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Bootstrap starting.
I20260806 20:42:41.828138 11409 log.cc:824] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:41.829056 11408 tablet_bootstrap.cc:492] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: No bootstrap required, opened a new log
I20260806 20:42:41.829339 11408 ts_tablet_manager.cc:1397] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260806 20:42:41.830833 11411 tablet_bootstrap.cc:654] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:41.831238 11409 tablet_bootstrap.cc:492] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: No bootstrap required, opened a new log
I20260806 20:42:41.831497 11409 ts_tablet_manager.cc:1397] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260806 20:42:41.832742 11411 log.cc:824] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:41.835718 11411 tablet_bootstrap.cc:492] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: No bootstrap required, opened a new log
I20260806 20:42:41.836002 11411 ts_tablet_manager.cc:1397] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 20:42:41.838485 11408 raft_consensus.cc:348] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.838902 11408 raft_consensus.cc:374] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:41.838999 11408 raft_consensus.cc:729] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e51055df69e403fb1ad58f903bcccd5, State: Initialized, Role: FOLLOWER
I20260806 20:42:41.839599 11409 raft_consensus.cc:348] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.839589 11408 consensus_queue.cc:260] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.839963 11409 raft_consensus.cc:374] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:41.840085 11409 raft_consensus.cc:729] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec3ff77e061d44a3b6154adfc0ce13aa, State: Initialized, Role: FOLLOWER
I20260806 20:42:41.840747 11409 consensus_queue.cc:260] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.841347 11408 ts_tablet_manager.cc:1428] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260806 20:42:41.842823 11252 heartbeater.cc:503] Master 127.23.200.62:36069 was elected leader, sending a full tablet report...
I20260806 20:42:41.843304 11409 ts_tablet_manager.cc:1428] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260806 20:42:41.846819 11411 raft_consensus.cc:348] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.847223 11411 raft_consensus.cc:374] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:41.847342 11411 raft_consensus.cc:729] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47c8c7396bc7413ba5916aa0f39fe01f, State: Initialized, Role: FOLLOWER
I20260806 20:42:41.847805 11411 consensus_queue.cc:260] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.848706 11390 heartbeater.cc:503] Master 127.23.200.62:36069 was elected leader, sending a full tablet report...
I20260806 20:42:41.849233 11411 ts_tablet_manager.cc:1428] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260806 20:42:41.866847 11417 raft_consensus.cc:482] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:41.867085 11417 raft_consensus.cc:504] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.868594 11417 leader_election.cc:288] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ec3ff77e061d44a3b6154adfc0ce13aa (127.23.200.2:43585), 8e51055df69e403fb1ad58f903bcccd5 (127.23.200.1:39413)
I20260806 20:42:41.875450 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.868956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:37729 (local address 127.23.200.2:43585)
0806 20:42:41.869155 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.869162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.869401 (+ 239us) server_negotiation.cc:407] Connection header received
0806 20:42:41.869500 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.869505 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.869589 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.869680 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:41.870140 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.871219 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.873460 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.874552 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.874716 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.874786 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.875053 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.875175 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:42:41.875447 11392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.868862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:43585 (local address 127.23.200.3:37729)
0806 20:42:41.869200 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.869220 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.869354 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.869747 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.869755 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.869788 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:41.870011 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.870020 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.871341 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.871345 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.873286 (+ 1941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.873301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.874703 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.874709 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.874891 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.874914 (+ 23us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.875035 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.875175 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260806 20:42:41.875792 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.869096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:35051 (local address 127.23.200.1:39413)
0806 20:42:41.869325 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.869334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.869802 (+ 468us) server_negotiation.cc:407] Connection header received
0806 20:42:41.869970 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.869974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.870028 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.870128 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:41.870681 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.871589 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.873866 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.874867 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.875054 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.875124 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.875409 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.875537 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:42:41.876120 11418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.868995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.23.200.3:35051)
0806 20:42:41.869714 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:41.869735 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:41.869852 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:41.870160 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:41.870163 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:41.870181 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:41.870536 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.870545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.871724 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.871734 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.873727 (+ 1993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:41.873740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.875068 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.875074 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:41.875240 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.875264 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 20:42:41.875372 (+ 108us) client_negotiation.cc:240] Negotiation successful
0806 20:42:41.875634 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"mutex_wait_us":1,"thread_start_us":383,"threads_started":1}
I20260806 20:42:41.876586 11206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0634d3f6c042359908328ace446657" candidate_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" is_pre_election: true
I20260806 20:42:41.877154 11206 raft_consensus.cc:2454] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47c8c7396bc7413ba5916aa0f39fe01f in term 0.
I20260806 20:42:41.877154 11071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0634d3f6c042359908328ace446657" candidate_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e51055df69e403fb1ad58f903bcccd5" is_pre_election: true
I20260806 20:42:41.877666 11071 raft_consensus.cc:2454] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47c8c7396bc7413ba5916aa0f39fe01f in term 0.
I20260806 20:42:41.877918 11277 leader_election.cc:302] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c8c7396bc7413ba5916aa0f39fe01f, ec3ff77e061d44a3b6154adfc0ce13aa; no voters:
I20260806 20:42:41.878360 11417 raft_consensus.cc:2781] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:42:41.878509 11417 raft_consensus.cc:482] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:42:41.878618 11417 raft_consensus.cc:3037] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:41.880746 11417 raft_consensus.cc:504] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.881354 11417 leader_election.cc:288] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [CANDIDATE]: Term 1 election: Requested vote from peers ec3ff77e061d44a3b6154adfc0ce13aa (127.23.200.2:43585), 8e51055df69e403fb1ad58f903bcccd5 (127.23.200.1:39413)
I20260806 20:42:41.881752 11071 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0634d3f6c042359908328ace446657" candidate_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e51055df69e403fb1ad58f903bcccd5"
I20260806 20:42:41.881752 11206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf0634d3f6c042359908328ace446657" candidate_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa"
I20260806 20:42:41.881961 11206 raft_consensus.cc:3037] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:41.881950 11071 raft_consensus.cc:3037] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:41.885141 11071 raft_consensus.cc:2454] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47c8c7396bc7413ba5916aa0f39fe01f in term 1.
I20260806 20:42:41.885141 11206 raft_consensus.cc:2454] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47c8c7396bc7413ba5916aa0f39fe01f in term 1.
I20260806 20:42:41.885542 11275 leader_election.cc:302] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47c8c7396bc7413ba5916aa0f39fe01f, 8e51055df69e403fb1ad58f903bcccd5; no voters:
I20260806 20:42:41.885841 11417 raft_consensus.cc:2781] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:41.886271 11417 raft_consensus.cc:686] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 1 LEADER]: Becoming Leader. State: Replica: 47c8c7396bc7413ba5916aa0f39fe01f, State: Running, Role: LEADER
I20260806 20:42:41.886749 11417 consensus_queue.cc:237] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } }
I20260806 20:42:41.891105 10924 catalog_manager.cc:5495] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47c8c7396bc7413ba5916aa0f39fe01f (127.23.200.3). New cstate: current_term: 1 leader_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47c8c7396bc7413ba5916aa0f39fe01f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 46815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 } health_report { overall_health: UNKNOWN } } }
I20260806 20:42:41.929814 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.924164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55646 (local address 127.23.200.3:46815)
0806 20:42:41.924359 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.924365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.924493 (+ 128us) server_negotiation.cc:407] Connection header received
0806 20:42:41.924760 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:41.924765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:41.924862 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:41.925021 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:42:41.925508 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.926231 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.927594 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:41.927997 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:41.928143 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:41.928727 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:42:41.929223 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:42:41.929412 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:41.929544 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 20:42:41.929591 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260806 20:42:41.931010 11117 tablet.cc:2369] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:41.938164 11071 raft_consensus.cc:1261] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 1 FOLLOWER]: Refusing update from remote peer 47c8c7396bc7413ba5916aa0f39fe01f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:41.938164 11206 raft_consensus.cc:1261] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 1 FOLLOWER]: Refusing update from remote peer 47c8c7396bc7413ba5916aa0f39fe01f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:41.939240 11419 consensus_queue.cc:1035] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e51055df69e403fb1ad58f903bcccd5" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:41.939625 11417 consensus_queue.cc:1035] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec3ff77e061d44a3b6154adfc0ce13aa" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:41.950634 11422 mvcc.cc:205] Tried to move back new op lower bound from 7315656548088070144 to 7315656547891957760. Current Snapshot: MvccSnapshot[applied={T|T < 7315656548088070144}]
I20260806 20:42:41.965124 10924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:41.965394 10924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:41.969051 10924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bbe26cf098a241f597cc50234f79df29: Sending DeleteTablet for 3 replicas of tablet bf0634d3f6c042359908328ace446657
I20260806 20:42:41.970214 11186 tablet_service.cc:1514] Processing DeleteTablet for tablet bf0634d3f6c042359908328ace446657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:41 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:60322
I20260806 20:42:41.970234 11051 tablet_service.cc:1514] Processing DeleteTablet for tablet bf0634d3f6c042359908328ace446657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:41 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:56488
I20260806 20:42:41.970412 11323 tablet_service.cc:1514] Processing DeleteTablet for tablet bf0634d3f6c042359908328ace446657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:41 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:55634
I20260806 20:42:41.977862 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10988
I20260806 20:42:42.007699 11431 tablet_replica.cc:330] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: stopping tablet replica
I20260806 20:42:42.008284 11431 raft_consensus.cc:2229] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:42.008961 11431 raft_consensus.cc:2258] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:42.009721 11433 tablet_replica.cc:330] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: stopping tablet replica
I20260806 20:42:42.010072 11433 raft_consensus.cc:2229] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:42:42.011472 11433 raft_consensus.cc:2258] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:42.013015 11432 tablet_replica.cc:330] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: stopping tablet replica
I20260806 20:42:42.014853 11431 ts_tablet_manager.cc:1905] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 20:42:42 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:42:42.016577 11433 ts_tablet_manager.cc:1905] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:42.019817 11432 raft_consensus.cc:2229] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:42.020429 11432 raft_consensus.cc:2258] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:42.021883 11431 ts_tablet_manager.cc:1918] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:42.022055 11431 log.cc:1196] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/bf0634d3f6c042359908328ace446657
I20260806 20:42:42.022621 11431 ts_tablet_manager.cc:1939] T bf0634d3f6c042359908328ace446657 P ec3ff77e061d44a3b6154adfc0ce13aa: Deleting consensus metadata
I20260806 20:42:42.023341 11433 ts_tablet_manager.cc:1918] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:42.023517 11433 log.cc:1196] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/bf0634d3f6c042359908328ace446657
I20260806 20:42:42.024124 11433 ts_tablet_manager.cc:1939] T bf0634d3f6c042359908328ace446657 P 47c8c7396bc7413ba5916aa0f39fe01f: Deleting consensus metadata
I20260806 20:42:42.024554 10912 catalog_manager.cc:4841] TS ec3ff77e061d44a3b6154adfc0ce13aa (127.23.200.2:43585): tablet bf0634d3f6c042359908328ace446657 (table test-table [id=53763c141b874689a30b0568fba71cff]) successfully deleted
I20260806 20:42:42.025943 10912 catalog_manager.cc:4841] TS 47c8c7396bc7413ba5916aa0f39fe01f (127.23.200.3:46815): tablet bf0634d3f6c042359908328ace446657 (table test-table [id=53763c141b874689a30b0568fba71cff]) successfully deleted
I20260806 20:42:42.028067 11432 ts_tablet_manager.cc:1905] T bf0634d3f6c042359908328ace446657 P 8e51055df69e403fb1ad58f903bcccd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:42.029171 11405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:41.979274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56494 (local address 127.23.200.1:39413)
0806 20:42:41.979515 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:41.979524 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:41.999808 (+ 20284us) server_negotiation.cc:407] Connection header received
0806 20:42:42.000360 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:42.000369 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:42.000526 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:42.000620 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:42.003050 (+ 2430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:42.004614 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:42.010881 (+ 6267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:42.011558 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:42.011712 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:42.020955 (+ 9243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:42.020971 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:42.020988 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:42.021033 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:42.025701 (+ 4668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:42.026677 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:42.026685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:42.026688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:42.026892 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:42.027328 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:42.027339 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:42.027342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:42.027764 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:42.028110 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:42.028490 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 20:42:42.028722 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:42:42.029923 11111 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:42.131549 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10988
W20260806 20:42:42.156213 10910 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:39413 (error 108)
I20260806 20:42:42.157052 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 11122
I20260806 20:42:42.158592 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.157877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56498)
0806 20:42:42.158209 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.158400 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
W20260806 20:42:42.158959 10910 catalog_manager.cc:4584] TS 8e51055df69e403fb1ad58f903bcccd5 (127.23.200.1:39413): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bf0634d3f6c042359908328ace446657: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Aug 06 20:42:42 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:42:42.176111 11403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.157993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60324 (local address 127.23.200.2:43585)
0806 20:42:42.158166 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:42.158173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:42.158351 (+ 178us) server_negotiation.cc:407] Connection header received
0806 20:42:42.158584 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:42.158588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:42.158692 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:42.158794 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:42.160757 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:42.161481 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:42.163691 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:42.164082 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:42.164178 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:42.169677 (+ 5499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:42.169691 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:42.169706 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:42.169738 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:42.172605 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:42.173456 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:42.173466 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:42.173471 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:42.173649 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:42.174194 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:42.174200 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:42.174203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:42.174785 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:42.175094 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:42.175511 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 20:42:42.175745 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 20:42:42.176770 11247 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:42.212543 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.211978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56502)
0806 20:42:42.212233 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.212317 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 20:42:42.213706 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.213306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56506)
0806 20:42:42.213488 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.213581 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 20:42:42.274480 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.273796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56522)
0806 20:42:42.274087 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.274220 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 20:42:42.275746 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.275350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56538)
0806 20:42:42.275522 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.275609 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 20:42:42.320051 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 11122
I20260806 20:42:42.339370 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 11258
Aug 06 20:42:42 dist-test-slave-tm1g krb5kdc[10880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:42:42.356516 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.356008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56548)
0806 20:42:42.356212 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.356310 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 20:42:42.357045 11407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.340251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.23.200.3:46815)
0806 20:42:42.340458 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:42.340464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:42.340489 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:42:42.340633 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:42.340648 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:42.340713 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:42.340832 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:42.342204 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:42.343126 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:42.344424 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:42.344892 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:42.345040 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:42.350185 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:42.350196 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:42.350211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:42.350242 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:42.353455 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:42.354429 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:42.354439 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:42.354445 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:42.354605 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:42.355134 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:42.355137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:42.355139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:42.355484 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:42.355842 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:42.356215 (+ 373us) server_negotiation.cc:299] Negotiation successful
0806 20:42:42.356433 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 20:42:42.357939 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.357574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56550)
0806 20:42:42.357761 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.357837 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 20:42:42.357940 11385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:42.478665 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 11258
I20260806 20:42:42.495915 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 10894
I20260806 20:42:42.497071 10956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:42.607499 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:42.606934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39413 (local address 127.0.0.1:56558)
0806 20:42:42.607181 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:42.607265 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:39413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 20:42:42.607456 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 10894
2026-08-06T20:42:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4337 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T20:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:42Z Disabled control of system clock
I20260806 20:42:42.686726 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36175
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 20:42:43.018039 11443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:43.018482 11443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:43.018633 11443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:43.025665 11443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:43.025755 11443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:43.025843 11443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:43.025946 11443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:43.026026 11443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:43.035068 11443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:36175
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:43.036288 11443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:43.038132 11443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:43.054080 11451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:43.054080 11448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:43.054076 11449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:43.055101 11443 server_base.cc:1034] running on GCE node
I20260806 20:42:43.055768 11443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:43.057049 11443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:43.058282 11443 hybrid_clock.cc:648] HybridClock initialized: now 1786048963058231 us; error 41 us; skew 500 ppm
I20260806 20:42:43.058668 11443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:43.060935 11443 webserver.cc:460] Webserver started at http://127.23.200.62:40841/ using document root <none> and password file <none>
I20260806 20:42:43.061605 11443 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:43.061736 11443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:43.062058 11443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:43.064884 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "6c23cc47414146e0afa460ec36aedd08"
format_stamp: "Formatted at 2026-08-06 20:42:43 on dist-test-slave-tm1g"
server_key: "b611d8631c5331b2965e3e5d1243081d"
server_key_iv: "4934d2a3564339ab96f35e5f59b73419"
server_key_version: "encryptionkey@0"
I20260806 20:42:43.065675 11443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "6c23cc47414146e0afa460ec36aedd08"
format_stamp: "Formatted at 2026-08-06 20:42:43 on dist-test-slave-tm1g"
server_key: "b611d8631c5331b2965e3e5d1243081d"
server_key_iv: "4934d2a3564339ab96f35e5f59b73419"
server_key_version: "encryptionkey@0"
I20260806 20:42:43.070577 11443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:42:43.073945 11458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:43.075304 11443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:43.075484 11443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "6c23cc47414146e0afa460ec36aedd08"
format_stamp: "Formatted at 2026-08-06 20:42:43 on dist-test-slave-tm1g"
server_key: "b611d8631c5331b2965e3e5d1243081d"
server_key_iv: "4934d2a3564339ab96f35e5f59b73419"
server_key_version: "encryptionkey@0"
I20260806 20:42:43.075691 11443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:43.088589 11443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:43.089325 11443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:43.089607 11443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:43.108372 11443 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:36175
I20260806 20:42:43.108394 11509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:36175 every 8 connection(s)
I20260806 20:42:43.109640 11443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:43.113240 11510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:43.118649 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 11443
I20260806 20:42:43.118839 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:43.119277 24352 external_mini_cluster.cc:1428] Setting key 9c3bf24936791b98bc74147738692237
I20260806 20:42:43.123896 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08: Bootstrap starting.
I20260806 20:42:43.126926 11510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:43.127890 11510 log.cc:824] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:43.130663 11510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08: No bootstrap required, opened a new log
I20260806 20:42:43.132294 11513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:43.121127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56010 (local address 127.23.200.62:36175)
0806 20:42:43.122094 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:43.122103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:43.122142 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:42:43.123811 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:43.123838 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:43.124288 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:43.124639 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:43.125192 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:43.126538 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:43.127919 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:43.128649 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:43.128889 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:43.130039 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:43.130047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:43.130060 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:43.130614 (+ 554us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:43.130698 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:43.130717 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:43.130851 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:43.131085 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 20:42:43.131377 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":336,"threads_started":1}
I20260806 20:42:43.138025 11510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c23cc47414146e0afa460ec36aedd08" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36175 } }
I20260806 20:42:43.138389 11510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:43.138499 11510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c23cc47414146e0afa460ec36aedd08, State: Initialized, Role: FOLLOWER
I20260806 20:42:43.139014 11510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c23cc47414146e0afa460ec36aedd08" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36175 } }
I20260806 20:42:43.139257 11510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:43.139400 11510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:43.139576 11510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:43.141633 11510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c23cc47414146e0afa460ec36aedd08" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36175 } }
I20260806 20:42:43.142139 11510 leader_election.cc:302] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c23cc47414146e0afa460ec36aedd08; no voters:
I20260806 20:42:43.142771 11510 leader_election.cc:288] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:43.142904 11515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:43.143666 11515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [term 1 LEADER]: Becoming Leader. State: Replica: 6c23cc47414146e0afa460ec36aedd08, State: Running, Role: LEADER
I20260806 20:42:43.144169 11515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c23cc47414146e0afa460ec36aedd08" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36175 } }
I20260806 20:42:43.144829 11510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:43.147127 11517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c23cc47414146e0afa460ec36aedd08. Latest consensus state: current_term: 1 leader_uuid: "6c23cc47414146e0afa460ec36aedd08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c23cc47414146e0afa460ec36aedd08" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36175 } } }
I20260806 20:42:43.147557 11517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:43.150754 11516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c23cc47414146e0afa460ec36aedd08" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c23cc47414146e0afa460ec36aedd08" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 36175 } } }
I20260806 20:42:43.151130 11516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:43.152573 11522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:43.157838 11522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:43.166028 11522 catalog_manager.cc:1347] Generated new cluster ID: 4161b35f1a5641b0a85bad9aaf23fbc7
I20260806 20:42:43.166142 11522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:43.182139 11522 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:43.183449 11522 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:43.194979 11522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08: Generated new TSK 0
I20260806 20:42:43.195909 11522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:42:43.213271 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36175
--builtin_ntp_servers=127.23.200.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 20:42:43.548101 11534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:43.548409 11534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:43.548568 11534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:43.555991 11534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:43.556123 11534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:43.556296 11534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:43.565198 11534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:43.566493 11534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:43.568250 11534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:43.581185 11542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:43.581243 11540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:43.581463 11539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:43.582324 11534 server_base.cc:1034] running on GCE node
I20260806 20:42:43.582937 11534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:43.583771 11534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:43.584976 11534 hybrid_clock.cc:648] HybridClock initialized: now 1786048963584940 us; error 24 us; skew 500 ppm
I20260806 20:42:43.585240 11534 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:43.587487 11534 webserver.cc:460] Webserver started at http://127.23.200.1:41697/ using document root <none> and password file <none>
I20260806 20:42:43.588105 11534 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:43.588191 11534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:43.588434 11534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:43.591183 11534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "02fce0b05c9b4cdb96d171467739bda2"
format_stamp: "Formatted at 2026-08-06 20:42:43 on dist-test-slave-tm1g"
server_key: "4c8ac2a1ad8f5e3cc77215719f18b77c"
server_key_iv: "8b6ec40e80b75086845e26b6f4bfee0b"
server_key_version: "encryptionkey@0"
I20260806 20:42:43.591898 11534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "02fce0b05c9b4cdb96d171467739bda2"
format_stamp: "Formatted at 2026-08-06 20:42:43 on dist-test-slave-tm1g"
server_key: "4c8ac2a1ad8f5e3cc77215719f18b77c"
server_key_iv: "8b6ec40e80b75086845e26b6f4bfee0b"
server_key_version: "encryptionkey@0"
I20260806 20:42:43.596647 11534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:42:43.599926 11549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:43.601226 11534 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:42:43.601359 11534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "02fce0b05c9b4cdb96d171467739bda2"
format_stamp: "Formatted at 2026-08-06 20:42:43 on dist-test-slave-tm1g"
server_key: "4c8ac2a1ad8f5e3cc77215719f18b77c"
server_key_iv: "8b6ec40e80b75086845e26b6f4bfee0b"
server_key_version: "encryptionkey@0"
I20260806 20:42:43.601531 11534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:43.617274 11534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:43.617952 11534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:43.618193 11534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:43.619134 11534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:43.620419 11534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:43.620517 11534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:43.620591 11534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:43.620645 11534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:43.653553 11534 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:45505
I20260806 20:42:43.653635 11664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:45505 every 8 connection(s)
I20260806 20:42:43.654925 11534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:43.661710 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 11534
I20260806 20:42:43.661876 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:42:43.662369 24352 external_mini_cluster.cc:1428] Setting key 66a0e88b87a57416ed583f5bb5329d56
I20260806 20:42:43.665215 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36175
--builtin_ntp_servers=127.23.200.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 20:42:43.668115 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:43.657382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:41191 (local address 127.23.200.62:36175)
0806 20:42:43.657829 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:43.657835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:43.658994 (+ 1159us) server_negotiation.cc:407] Connection header received
0806 20:42:43.660794 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:43.660801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:43.660896 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:43.661017 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:43.662121 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:43.663002 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:43.665015 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:43.665739 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:43.665887 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:43.667162 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:43.667170 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:43.667177 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:43.667201 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:43.667255 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:43.667272 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:43.667367 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:43.667566 (+ 199us) server_negotiation.cc:299] Negotiation successful
0806 20:42:43.667671 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":213,"threads_started":1}
I20260806 20:42:43.668784 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:43.657957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36175 (local address 127.23.200.1:41191)
0806 20:42:43.658834 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:43.658870 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:43.660536 (+ 1666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:43.661247 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:43.661255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:43.661336 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:43.661956 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:43.661983 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:43.663207 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:43.663212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:43.664866 (+ 1654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:43.664878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:43.665946 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:43.665952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:43.666139 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:43.666881 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:43.666884 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:43.666920 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:43.666988 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:43.667370 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:43.667378 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:43.667383 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 20:42:43.667511 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 20:42:43.667770 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":314,"threads_started":1}
I20260806 20:42:43.670192 11665 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36175
I20260806 20:42:43.670498 11665 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:43.671190 11665 heartbeater.cc:511] Master 127.23.200.62:36175 requested a full tablet report, sending...
I20260806 20:42:43.673131 11475 ts_manager.cc:194] Registered new tserver with Master: 02fce0b05c9b4cdb96d171467739bda2 (127.23.200.1:45505)
I20260806 20:42:43.675200 11475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.1:41191
W20260806 20:42:43.996562 11669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:43.996882 11669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:43.997063 11669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:44.004174 11669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:44.004297 11669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:44.004487 11669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:44.013505 11669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:44.014855 11669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:44.016620 11669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:44.029706 11674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:44.029680 11675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:44.029740 11677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:44.030532 11669 server_base.cc:1034] running on GCE node
I20260806 20:42:44.031082 11669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:44.032042 11669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:44.033267 11669 hybrid_clock.cc:648] HybridClock initialized: now 1786048964033232 us; error 48 us; skew 500 ppm
I20260806 20:42:44.033571 11669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:44.035697 11669 webserver.cc:460] Webserver started at http://127.23.200.2:35747/ using document root <none> and password file <none>
I20260806 20:42:44.036360 11669 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:44.036485 11669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:44.036794 11669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:44.039613 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d"
format_stamp: "Formatted at 2026-08-06 20:42:44 on dist-test-slave-tm1g"
server_key: "7c344f447e2dd90bf24ac019e4b87b15"
server_key_iv: "6c5eabc3a896dc554eb0d2d5120ff2d9"
server_key_version: "encryptionkey@0"
I20260806 20:42:44.040382 11669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d"
format_stamp: "Formatted at 2026-08-06 20:42:44 on dist-test-slave-tm1g"
server_key: "7c344f447e2dd90bf24ac019e4b87b15"
server_key_iv: "6c5eabc3a896dc554eb0d2d5120ff2d9"
server_key_version: "encryptionkey@0"
I20260806 20:42:44.045468 11669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:42:44.048810 11683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:44.050014 11669 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 20:42:44.050192 11669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d"
format_stamp: "Formatted at 2026-08-06 20:42:44 on dist-test-slave-tm1g"
server_key: "7c344f447e2dd90bf24ac019e4b87b15"
server_key_iv: "6c5eabc3a896dc554eb0d2d5120ff2d9"
server_key_version: "encryptionkey@0"
I20260806 20:42:44.050470 11669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:44.078531 11669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:44.079505 11669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:44.079828 11669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:44.080822 11669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:44.082286 11669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:44.082429 11669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:44.082588 11669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:44.082677 11669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:44.116511 11669 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:37787
I20260806 20:42:44.116569 11798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:37787 every 8 connection(s)
I20260806 20:42:44.117763 11669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:44.122964 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 11669
I20260806 20:42:44.123159 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:42:44.123641 24352 external_mini_cluster.cc:1428] Setting key 561e656e5407f321d860ea33ce92513f
I20260806 20:42:44.126672 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36175
--builtin_ntp_servers=127.23.200.20:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 20:42:44.132134 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.120373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:41989 (local address 127.23.200.62:36175)
0806 20:42:44.120603 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.120610 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.122026 (+ 1416us) server_negotiation.cc:407] Connection header received
0806 20:42:44.124527 (+ 2501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.124533 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.124619 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.124739 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:44.126023 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.126978 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.129426 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.129797 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.129888 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.131420 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:44.131424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:44.131427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:44.131444 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:44.131473 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:44.131484 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:44.131554 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.131819 (+ 265us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.131902 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 20:42:44.133527 11801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.120839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36175 (local address 127.23.200.2:41989)
0806 20:42:44.121865 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.121914 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.124260 (+ 2346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.124985 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.124992 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.125076 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:44.125832 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.125858 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.127141 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.127150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.129289 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.129307 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.130011 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.130020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.130354 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.131092 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:44.131098 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:44.131146 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:44.131239 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:44.131583 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:44.131595 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:44.131607 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.131801 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.132199 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":301,"threads_started":1}
I20260806 20:42:44.134902 11799 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36175
I20260806 20:42:44.135198 11799 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:44.135862 11799 heartbeater.cc:511] Master 127.23.200.62:36175 requested a full tablet report, sending...
I20260806 20:42:44.137284 11475 ts_manager.cc:194] Registered new tserver with Master: e8c1c3da219a428e8f52c8cc1b7ae88d (127.23.200.2:37787)
I20260806 20:42:44.138067 11475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.2:41989
W20260806 20:42:44.457118 11802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:44.457371 11802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:44.457520 11802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:44.465085 11802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:44.465174 11802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:44.465358 11802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:44.474067 11802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:44.475270 11802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:44.476953 11802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:44.491323 11808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:44.491416 11807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:44.491437 11810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:44.492401 11802 server_base.cc:1034] running on GCE node
I20260806 20:42:44.493018 11802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:44.493854 11802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:44.495052 11802 hybrid_clock.cc:648] HybridClock initialized: now 1786048964494996 us; error 51 us; skew 500 ppm
I20260806 20:42:44.495361 11802 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:44.497584 11802 webserver.cc:460] Webserver started at http://127.23.200.3:44315/ using document root <none> and password file <none>
I20260806 20:42:44.498251 11802 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:44.498416 11802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:44.498744 11802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:44.501430 11802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "2410a7fc1a264f2db779589802acdc78"
format_stamp: "Formatted at 2026-08-06 20:42:44 on dist-test-slave-tm1g"
server_key: "8266eab923eb46f24a35485ebaa500b4"
server_key_iv: "262a32e1e22afb23de56609b66f62934"
server_key_version: "encryptionkey@0"
I20260806 20:42:44.502194 11802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "2410a7fc1a264f2db779589802acdc78"
format_stamp: "Formatted at 2026-08-06 20:42:44 on dist-test-slave-tm1g"
server_key: "8266eab923eb46f24a35485ebaa500b4"
server_key_iv: "262a32e1e22afb23de56609b66f62934"
server_key_version: "encryptionkey@0"
I20260806 20:42:44.507176 11802 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260806 20:42:44.510475 11816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:44.511703 11802 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:42:44.511876 11802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "2410a7fc1a264f2db779589802acdc78"
format_stamp: "Formatted at 2026-08-06 20:42:44 on dist-test-slave-tm1g"
server_key: "8266eab923eb46f24a35485ebaa500b4"
server_key_iv: "262a32e1e22afb23de56609b66f62934"
server_key_version: "encryptionkey@0"
I20260806 20:42:44.512084 11802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:44.532287 11802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:44.533039 11802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:44.533337 11802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:44.534338 11802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:44.535777 11802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:44.535897 11802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:44.536036 11802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:44.536125 11802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:44.569339 11802 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:35187
I20260806 20:42:44.569440 11930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:35187 every 8 connection(s)
I20260806 20:42:44.570662 11802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:44.575642 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 11802
I20260806 20:42:44.575840 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:42:44.576411 24352 external_mini_cluster.cc:1428] Setting key a84cc09309c16cd8601f6274908f2a9e
I20260806 20:42:44.583841 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.573185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:34271 (local address 127.23.200.62:36175)
0806 20:42:44.573401 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.573409 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.574831 (+ 1422us) server_negotiation.cc:407] Connection header received
0806 20:42:44.577236 (+ 2405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.577240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.577312 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.577393 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:44.578775 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.579816 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.581392 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.581810 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.581900 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.583164 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:44.583169 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:44.583171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:44.583187 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:44.583216 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:44.583227 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:44.583291 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.583508 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.583582 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:42:44.584694 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.573752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:36175 (local address 127.23.200.3:34271)
0806 20:42:44.574675 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.574712 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.576937 (+ 2225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.577643 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.577653 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.577768 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:44.578619 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.578639 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.579955 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.579959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.581258 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.581270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.581980 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.581985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.582166 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.582888 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:44.582891 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:44.582927 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:44.583002 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:44.583322 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:44.583330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:44.583335 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.583465 (+ 130us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.583723 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":267,"threads_started":1}
I20260806 20:42:44.585916 11931 heartbeater.cc:347] Connected to a master server at 127.23.200.62:36175
I20260806 20:42:44.586193 11931 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:44.586911 11931 heartbeater.cc:511] Master 127.23.200.62:36175 requested a full tablet report, sending...
I20260806 20:42:44.588212 11475 ts_manager.cc:194] Registered new tserver with Master: 2410a7fc1a264f2db779589802acdc78 (127.23.200.3:35187)
I20260806 20:42:44.589012 11475 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.3:34271
I20260806 20:42:44.591859 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:42:44.601380 11667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.595909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56018 (local address 127.23.200.62:36175)
0806 20:42:44.596052 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.596056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.596528 (+ 472us) server_negotiation.cc:407] Connection header received
0806 20:42:44.596644 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.596647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.596710 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.596786 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:44.597199 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.597877 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.599241 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.599671 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.599766 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.600784 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:44.600789 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:44.600792 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:44.600816 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:44.600846 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:44.600857 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:44.600964 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.601125 (+ 161us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.601167 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:42:44.605087 11475 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:42:44.611948 11475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:42:44.655938 11943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.623238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:37787 (local address 127.0.0.1:50998)
0806 20:42:44.644773 (+ 21535us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.644840 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.645054 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.647544 (+ 2490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.647552 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.647583 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.648198 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.648215 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.650341 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.650347 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.652973 (+ 2626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.652988 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.655128 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.655134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.655322 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.655357 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.655444 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.655577 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19406,"mutex_wait_us":109,"thread_start_us":19242,"threads_started":1}
I20260806 20:42:44.656589 11946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.623462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:35187 (local address 127.0.0.1:33714)
0806 20:42:44.646757 (+ 23295us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.646780 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.647377 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.648394 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.648400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.648438 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.648811 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.648827 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.650630 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.650638 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.652590 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.652608 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.654885 (+ 2277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.654892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.655168 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.655218 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.655338 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.655515 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23106,"mutex_wait_us":103,"thread_start_us":24391,"threads_started":1}
I20260806 20:42:44.657040 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.623509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50998 (local address 127.23.200.2:37787)
0806 20:42:44.647049 (+ 23540us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.647054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.647079 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:42:44.647155 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.647160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.647341 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.647547 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.648489 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.650176 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.653196 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.654656 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.654772 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.654989 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.656548 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.656712 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23390,"thread_start_us":23293,"threads_started":1}
I20260806 20:42:44.657531 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.624036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33714 (local address 127.23.200.3:35187)
0806 20:42:44.645469 (+ 21433us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.645492 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.647566 (+ 2074us) server_negotiation.cc:407] Connection header received
0806 20:42:44.647694 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.647704 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.647957 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.648139 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.649030 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.650499 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.652851 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.654295 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.654460 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.654754 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.655709 (+ 955us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.655888 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21182,"thread_start_us":21462,"threads_started":1}
I20260806 20:42:44.658628 11945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.623863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54160)
0806 20:42:44.648235 (+ 24372us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.648257 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.648379 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.648903 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.648909 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.648933 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.649297 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.649312 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.651457 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.651463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.653713 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.653727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.656228 (+ 2501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.656236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.658011 (+ 1775us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.658051 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.658153 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.658271 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24240,"mutex_wait_us":23,"thread_start_us":22536,"threads_started":1}
I20260806 20:42:44.658797 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.624406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.23.200.1:45505)
0806 20:42:44.646194 (+ 21788us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.646216 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.648379 (+ 2163us) server_negotiation.cc:407] Connection header received
0806 20:42:44.648586 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.648591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.648726 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.648913 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.649729 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.651245 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.653911 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.655685 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.655814 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.656107 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.658236 (+ 2129us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.658462 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21595,"thread_start_us":21515,"threads_started":1}
I20260806 20:42:44.660387 11732 tablet_service.cc:1467] Processing CreateTablet for tablet 126a95859d8e462da2970c6123d65812 (DEFAULT_TABLE table=test-table [id=10de6a7e6df846cba67bca06baa71e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:44.661775 11732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126a95859d8e462da2970c6123d65812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:44.663386 11598 tablet_service.cc:1467] Processing CreateTablet for tablet 126a95859d8e462da2970c6123d65812 (DEFAULT_TABLE table=test-table [id=10de6a7e6df846cba67bca06baa71e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:44.663398 11865 tablet_service.cc:1467] Processing CreateTablet for tablet 126a95859d8e462da2970c6123d65812 (DEFAULT_TABLE table=test-table [id=10de6a7e6df846cba67bca06baa71e5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:44.664489 11598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126a95859d8e462da2970c6123d65812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:44.664741 11865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 126a95859d8e462da2970c6123d65812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:44.674753 11949 tablet_bootstrap.cc:492] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: Bootstrap starting.
I20260806 20:42:44.678565 11665 heartbeater.cc:503] Master 127.23.200.62:36175 was elected leader, sending a full tablet report...
I20260806 20:42:44.679126 11949 tablet_bootstrap.cc:654] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:44.680526 11951 tablet_bootstrap.cc:492] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Bootstrap starting.
I20260806 20:42:44.682108 11950 tablet_bootstrap.cc:492] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Bootstrap starting.
I20260806 20:42:44.684073 11951 tablet_bootstrap.cc:654] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:44.684213 11949 log.cc:824] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:44.685273 11951 log.cc:824] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:44.685463 11950 tablet_bootstrap.cc:654] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:44.686722 11950 log.cc:824] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:44.687388 11949 tablet_bootstrap.cc:492] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: No bootstrap required, opened a new log
I20260806 20:42:44.687677 11949 ts_tablet_manager.cc:1397] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260806 20:42:44.688344 11951 tablet_bootstrap.cc:492] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: No bootstrap required, opened a new log
I20260806 20:42:44.688830 11951 ts_tablet_manager.cc:1397] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260806 20:42:44.689652 11950 tablet_bootstrap.cc:492] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: No bootstrap required, opened a new log
I20260806 20:42:44.689893 11950 ts_tablet_manager.cc:1397] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260806 20:42:44.696856 11950 raft_consensus.cc:348] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.697225 11950 raft_consensus.cc:374] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:44.697384 11950 raft_consensus.cc:729] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8c1c3da219a428e8f52c8cc1b7ae88d, State: Initialized, Role: FOLLOWER
I20260806 20:42:44.697997 11950 consensus_queue.cc:260] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.699143 11799 heartbeater.cc:503] Master 127.23.200.62:36175 was elected leader, sending a full tablet report...
I20260806 20:42:44.699129 11951 raft_consensus.cc:348] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.699522 11951 raft_consensus.cc:374] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:44.699623 11951 raft_consensus.cc:729] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2410a7fc1a264f2db779589802acdc78, State: Initialized, Role: FOLLOWER
I20260806 20:42:44.699730 11950 ts_tablet_manager.cc:1428] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20260806 20:42:44.698657 11949 raft_consensus.cc:348] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.700296 11951 consensus_queue.cc:260] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.700604 11949 raft_consensus.cc:374] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:44.700704 11949 raft_consensus.cc:729] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02fce0b05c9b4cdb96d171467739bda2, State: Initialized, Role: FOLLOWER
I20260806 20:42:44.701372 11949 consensus_queue.cc:260] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.701936 11931 heartbeater.cc:503] Master 127.23.200.62:36175 was elected leader, sending a full tablet report...
I20260806 20:42:44.702028 11951 ts_tablet_manager.cc:1428] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260806 20:42:44.704627 11949 ts_tablet_manager.cc:1428] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: Time spent starting tablet: real 0.017s user 0.008s sys 0.007s
W20260806 20:42:44.822499 11932 tablet.cc:2369] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:42:44.869906 11800 tablet.cc:2369] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:42:44.907236 11666 tablet.cc:2369] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:44.931301 11956 raft_consensus.cc:482] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:44.931651 11956 raft_consensus.cc:504] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.933314 11956 leader_election.cc:288] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8c1c3da219a428e8f52c8cc1b7ae88d (127.23.200.2:37787), 02fce0b05c9b4cdb96d171467739bda2 (127.23.200.1:45505)
I20260806 20:42:44.935608 11957 raft_consensus.cc:482] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:44.935932 11957 raft_consensus.cc:504] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.941397 11933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.933806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.23.200.3:53535)
0806 20:42:44.934155 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.934195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.934372 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.934821 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.934827 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.934856 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.935042 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.935051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.936461 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.936467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.938226 (+ 1759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.938243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.940399 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.940407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.940636 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.940688 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.940784 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.940978 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235}
I20260806 20:42:44.942221 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.933855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:53535 (local address 127.23.200.1:45505)
0806 20:42:44.934088 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.934094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.934288 (+ 194us) server_negotiation.cc:407] Connection header received
0806 20:42:44.934553 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.934560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.934642 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.934766 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.935197 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.936311 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.938492 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.940020 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.940201 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.940302 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.941702 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.941878 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 20:42:44.942546 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.933788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:54601 (local address 127.23.200.2:37787)
0806 20:42:44.934018 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.934024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.934705 (+ 681us) server_negotiation.cc:407] Connection header received
0806 20:42:44.934830 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.934835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.934920 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.934999 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.935683 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.936619 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.939319 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.941045 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.941198 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.941296 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.941959 (+ 663us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.942160 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:42:44.943490 11958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.933855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:37787 (local address 127.23.200.3:54601)
0806 20:42:44.934608 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.934620 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.934702 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.935049 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.935053 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.935080 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.935510 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.935520 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.936791 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.936796 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.939056 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.939075 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.941484 (+ 2409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.941492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.941736 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.941784 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.942877 (+ 1093us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.943070 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":476,"threads_started":1}
I20260806 20:42:44.943482 11619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126a95859d8e462da2970c6123d65812" candidate_uuid: "2410a7fc1a264f2db779589802acdc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fce0b05c9b4cdb96d171467739bda2" is_pre_election: true
I20260806 20:42:44.943923 11957 leader_election.cc:288] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8c1c3da219a428e8f52c8cc1b7ae88d (127.23.200.2:37787), 2410a7fc1a264f2db779589802acdc78 (127.23.200.3:35187)
I20260806 20:42:44.943964 11619 raft_consensus.cc:2454] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2410a7fc1a264f2db779589802acdc78 in term 0.
I20260806 20:42:44.944783 11818 leader_election.cc:302] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02fce0b05c9b4cdb96d171467739bda2, 2410a7fc1a264f2db779589802acdc78; no voters:
I20260806 20:42:44.944782 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126a95859d8e462da2970c6123d65812" candidate_uuid: "2410a7fc1a264f2db779589802acdc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" is_pre_election: true
I20260806 20:42:44.945206 11956 raft_consensus.cc:2781] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:42:44.945374 11956 raft_consensus.cc:482] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:42:44.945363 11752 raft_consensus.cc:2454] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2410a7fc1a264f2db779589802acdc78 in term 0.
I20260806 20:42:44.945502 11956 raft_consensus.cc:3037] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:44.948354 11956 raft_consensus.cc:504] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.948999 11956 leader_election.cc:288] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [CANDIDATE]: Term 1 election: Requested vote from peers e8c1c3da219a428e8f52c8cc1b7ae88d (127.23.200.2:37787), 02fce0b05c9b4cdb96d171467739bda2 (127.23.200.1:45505)
I20260806 20:42:44.949414 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126a95859d8e462da2970c6123d65812" candidate_uuid: "2410a7fc1a264f2db779589802acdc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d"
I20260806 20:42:44.949615 11752 raft_consensus.cc:3037] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:44.949616 11619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126a95859d8e462da2970c6123d65812" candidate_uuid: "2410a7fc1a264f2db779589802acdc78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02fce0b05c9b4cdb96d171467739bda2"
I20260806 20:42:44.949832 11619 raft_consensus.cc:3037] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:44.952553 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.938365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:56753 (local address 127.23.200.3:35187)
0806 20:42:44.938543 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.938551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.944573 (+ 6022us) server_negotiation.cc:407] Connection header received
0806 20:42:44.944850 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.944855 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.944939 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.945001 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.946096 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.947242 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.950083 (+ 2841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.951308 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.951443 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.951511 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.952212 (+ 701us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.952334 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260806 20:42:44.952631 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.938663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49857 (local address 127.23.200.2:37787)
0806 20:42:44.942818 (+ 4155us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.942826 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.945375 (+ 2549us) server_negotiation.cc:407] Connection header received
0806 20:42:44.945658 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.945664 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.945760 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.945828 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:44.946641 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.947398 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.950087 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.951444 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.951606 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.951682 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:44.952212 (+ 530us) server_negotiation.cc:299] Negotiation successful
0806 20:42:44.952334 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4029,"thread_start_us":5929,"threads_started":1}
I20260806 20:42:44.953063 11752 raft_consensus.cc:2454] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2410a7fc1a264f2db779589802acdc78 in term 1.
I20260806 20:42:44.953428 11960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.938024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:35187 (local address 127.23.200.1:56753)
0806 20:42:44.944482 (+ 6458us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.944506 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.944675 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.945365 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.945372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.945402 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.945912 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.945923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.947386 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.947391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.949833 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.949845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.951689 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.951698 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.951963 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.952008 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.952708 (+ 700us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.952852 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6325,"mutex_wait_us":53,"thread_start_us":8235,"threads_started":1}
I20260806 20:42:44.953433 11619 raft_consensus.cc:2454] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2410a7fc1a264f2db779589802acdc78 in term 1.
I20260806 20:42:44.953756 11959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.938086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:37787 (local address 127.23.200.1:49857)
0806 20:42:44.945264 (+ 7178us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:44.945282 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:44.945482 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:44.946048 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:44.946052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:44.946075 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:44.946466 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.946480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.947578 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.947583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.949830 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:44.949845 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.951864 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.951869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:44.952048 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:44.952076 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 20:42:44.952926 (+ 850us) client_negotiation.cc:240] Negotiation successful
0806 20:42:44.953043 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7036,"mutex_wait_us":73,"thread_start_us":6578,"threads_started":1}
I20260806 20:42:44.953774 11820 leader_election.cc:302] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2410a7fc1a264f2db779589802acdc78, e8c1c3da219a428e8f52c8cc1b7ae88d; no voters:
I20260806 20:42:44.954087 11956 raft_consensus.cc:2781] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:44.954389 11886 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126a95859d8e462da2970c6123d65812" candidate_uuid: "02fce0b05c9b4cdb96d171467739bda2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2410a7fc1a264f2db779589802acdc78" is_pre_election: true
I20260806 20:42:44.954484 11752 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "126a95859d8e462da2970c6123d65812" candidate_uuid: "02fce0b05c9b4cdb96d171467739bda2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" is_pre_election: true
I20260806 20:42:44.954919 11752 raft_consensus.cc:2379] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 02fce0b05c9b4cdb96d171467739bda2 in current term 1: Already voted for candidate 2410a7fc1a264f2db779589802acdc78 in this term.
I20260806 20:42:44.955655 11956 raft_consensus.cc:686] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 1 LEADER]: Becoming Leader. State: Replica: 2410a7fc1a264f2db779589802acdc78, State: Running, Role: LEADER
I20260806 20:42:44.956146 11956 consensus_queue.cc:237] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } }
I20260806 20:42:44.959949 11550 leader_election.cc:302] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 02fce0b05c9b4cdb96d171467739bda2; no voters: 2410a7fc1a264f2db779589802acdc78, e8c1c3da219a428e8f52c8cc1b7ae88d
I20260806 20:42:44.960331 11957 raft_consensus.cc:2726] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 20:42:44.960989 11473 catalog_manager.cc:5495] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2410a7fc1a264f2db779589802acdc78 (127.23.200.3). New cstate: current_term: 1 leader_uuid: "2410a7fc1a264f2db779589802acdc78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2410a7fc1a264f2db779589802acdc78" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 35187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 } health_report { overall_health: UNKNOWN } } }
I20260806 20:42:45.002216 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:44.996777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33730 (local address 127.23.200.3:35187)
0806 20:42:44.996958 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:44.996964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:44.996982 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:42:44.997220 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:44.997224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:44.997296 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:44.997450 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:42:44.997827 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:44.998630 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:44.999991 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.000392 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.000525 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:45.001114 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:42:45.001580 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:42:45.001759 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:45.001912 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 20:42:45.001954 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260806 20:42:45.010111 11619 raft_consensus.cc:1261] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 1 FOLLOWER]: Refusing update from remote peer 2410a7fc1a264f2db779589802acdc78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:45.010263 11752 raft_consensus.cc:1261] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 1 FOLLOWER]: Refusing update from remote peer 2410a7fc1a264f2db779589802acdc78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:45.011020 11962 consensus_queue.cc:1035] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02fce0b05c9b4cdb96d171467739bda2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 45505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:45.011468 11956 consensus_queue.cc:1035] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8c1c3da219a428e8f52c8cc1b7ae88d" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 37787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:45.025173 11967 mvcc.cc:205] Tried to move back new op lower bound from 7315656560672186368 to 7315656560463491072. Current Snapshot: MvccSnapshot[applied={T|T < 7315656560672186368}]
I20260806 20:42:45.035352 11475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:45.035620 11475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:45.039292 11475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6c23cc47414146e0afa460ec36aedd08: Sending DeleteTablet for 3 replicas of tablet 126a95859d8e462da2970c6123d65812
I20260806 20:42:45.040385 11865 tablet_service.cc:1514] Processing DeleteTablet for tablet 126a95859d8e462da2970c6123d65812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:45 UTC) from {username='slave'} at 127.0.0.1:33714
I20260806 20:42:45.040488 11598 tablet_service.cc:1514] Processing DeleteTablet for tablet 126a95859d8e462da2970c6123d65812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:45 UTC) from {username='slave'} at 127.0.0.1:54160
I20260806 20:42:45.040602 11732 tablet_service.cc:1514] Processing DeleteTablet for tablet 126a95859d8e462da2970c6123d65812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:45 UTC) from {username='slave'} at 127.0.0.1:50998
I20260806 20:42:45.041249 11975 tablet_replica.cc:330] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: stopping tablet replica
I20260806 20:42:45.041713 11975 raft_consensus.cc:2229] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:42:45.042403 11975 raft_consensus.cc:2258] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:45.042476 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 11534
I20260806 20:42:45.053125 11977 tablet_replica.cc:330] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2: stopping tablet replica
I20260806 20:42:45.054105 11976 tablet_replica.cc:330] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: stopping tablet replica
I20260806 20:42:45.054530 11976 raft_consensus.cc:2229] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:45.054831 11975 ts_tablet_manager.cc:1905] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:45.053573 11977 raft_consensus.cc:2229] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:45.055076 11948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.043479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.23.200.1:45505)
0806 20:42:45.043645 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:45.043651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:45.046121 (+ 2470us) server_negotiation.cc:407] Connection header received
0806 20:42:45.046330 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:45.046336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:45.046469 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:45.046572 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:45.047197 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.048464 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.050802 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.051616 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.051733 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:45.053297 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:45.053309 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:45.053325 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:45.054070 (+ 745us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:45.054128 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:45.054153 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:45.054241 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:45.054535 (+ 294us) server_negotiation.cc:299] Negotiation successful
0806 20:42:45.054650 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260806 20:42:45.055326 11977 raft_consensus.cc:2258] T 126a95859d8e462da2970c6123d65812 P 02fce0b05c9b4cdb96d171467739bda2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:45.056175 11660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:45.061956 11976 raft_consensus.cc:2258] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:45.065805 11975 ts_tablet_manager.cc:1918] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:45.065950 11976 ts_tablet_manager.cc:1905] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:45.065976 11975 log.cc:1196] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/126a95859d8e462da2970c6123d65812
I20260806 20:42:45.066598 11975 ts_tablet_manager.cc:1939] T 126a95859d8e462da2970c6123d65812 P 2410a7fc1a264f2db779589802acdc78: Deleting consensus metadata
I20260806 20:42:45.068418 11459 catalog_manager.cc:4841] TS 2410a7fc1a264f2db779589802acdc78 (127.23.200.3:35187): tablet 126a95859d8e462da2970c6123d65812 (table test-table [id=10de6a7e6df846cba67bca06baa71e5c]) successfully deleted
I20260806 20:42:45.073128 11976 ts_tablet_manager.cc:1918] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:45.073299 11976 log.cc:1196] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/126a95859d8e462da2970c6123d65812
I20260806 20:42:45.073860 11976 ts_tablet_manager.cc:1939] T 126a95859d8e462da2970c6123d65812 P e8c1c3da219a428e8f52c8cc1b7ae88d: Deleting consensus metadata
I20260806 20:42:45.075799 11462 catalog_manager.cc:4841] TS e8c1c3da219a428e8f52c8cc1b7ae88d (127.23.200.2:37787): tablet 126a95859d8e462da2970c6123d65812 (table test-table [id=10de6a7e6df846cba67bca06baa71e5c]) successfully deleted
I20260806 20:42:45.175508 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 11534
W20260806 20:42:45.195703 11460 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.200.1:45505 (error 108)
I20260806 20:42:45.195925 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 11669
I20260806 20:42:45.198213 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.197353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54180)
0806 20:42:45.197804 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.197986 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":235,"threads_started":1}
W20260806 20:42:45.198585 11460 catalog_manager.cc:4584] TS 02fce0b05c9b4cdb96d171467739bda2 (127.23.200.1:45505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 126a95859d8e462da2970c6123d65812: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
I20260806 20:42:45.204137 11944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.197111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51006 (local address 127.23.200.2:37787)
0806 20:42:45.197289 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:45.197293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:45.197309 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:42:45.197364 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:45.197367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:45.197453 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:45.197589 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:45.198251 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.199442 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.200876 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.201540 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.201676 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:45.202821 (+ 1145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:45.202829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:45.202843 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:45.203410 (+ 567us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:45.203472 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:45.203491 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:45.203577 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:45.203732 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 20:42:45.203811 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 20:42:45.204787 11794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:45.252063 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.251621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54182)
0806 20:42:45.251832 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.251913 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 20:42:45.253144 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.252761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54198)
0806 20:42:45.252928 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.253010 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 20:42:45.313783 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.313320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54210)
0806 20:42:45.313549 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.313636 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 20:42:45.314947 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.314549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54212)
0806 20:42:45.314708 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.314799 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260806 20:42:45.355363 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 11669
I20260806 20:42:45.374588 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 11802
I20260806 20:42:45.382644 11947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.375633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33734 (local address 127.23.200.3:35187)
0806 20:42:45.375800 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:45.375804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:45.375822 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:42:45.375943 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:45.375947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:45.376044 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:45.376134 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:45.376651 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.377404 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.379380 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:45.379849 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:45.380008 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:45.381140 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:45.381148 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:45.381163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:45.381801 (+ 638us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:45.381852 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:45.381871 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:45.381947 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:45.382141 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 20:42:45.382218 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 20:42:45.383455 11926 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:45.395612 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.395172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54228)
0806 20:42:45.395373 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.395466 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 20:42:45.396760 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.396359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54244)
0806 20:42:45.396568 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.396640 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 20:42:45.508396 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 11802
I20260806 20:42:45.529104 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 11443
I20260806 20:42:45.530180 11505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:45.638734 11981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:45.638024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:45505 (local address 127.0.0.1:54258)
0806 20:42:45.638287 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:45.638517 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.1:45505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 20:42:45.639027 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 11443
2026-08-06T20:42:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3030 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:45 dist-test-slave-tm1g krb5kdc[11987](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:45 dist-test-slave-tm1g krb5kdc[11987](info): set up 2 sockets
Aug 06 20:42:45 dist-test-slave-tm1g krb5kdc[11987](info): commencing operation
krb5kdc: starting...
W20260806 20:42:46.761897 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:42:46 dist-test-slave-tm1g krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:42:47.011417 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:39471
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:39471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:47.340468 12000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:47.340808 12000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:47.340983 12000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:47.348172 12000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:47.348325 12000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:47.348434 12000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:47.348542 12000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:47.348611 12000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:47.358142 12000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:39471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:39471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:47.359530 12000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:47.360843 12000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:47.372403 12006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:47.372453 12007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:47.372463 12009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:47.373828 12000 server_base.cc:1034] running on GCE node
I20260806 20:42:47.374660 12000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:47.376017 12000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:47.377247 12000 hybrid_clock.cc:648] HybridClock initialized: now 1786048967377211 us; error 44 us; skew 500 ppm
I20260806 20:42:47.377585 12000 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:47 dist-test-slave-tm1g krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048967, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:47.385390 12000 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:42:47.387004 12000 webserver.cc:460] Webserver started at http://127.23.200.62:41773/ using document root <none> and password file <none>
I20260806 20:42:47.387705 12000 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:47.387853 12000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:47.388168 12000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:47.391244 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "9bfca7b285074417a0772d8b2eeaa32a"
format_stamp: "Formatted at 2026-08-06 20:42:47 on dist-test-slave-tm1g"
server_key: "4e91ba51cadcbb69690baee1fb4c7b14"
server_key_iv: "5c2b78c770d924a22aff41ffba6ef3ee"
server_key_version: "encryptionkey@0"
I20260806 20:42:47.391991 12000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "9bfca7b285074417a0772d8b2eeaa32a"
format_stamp: "Formatted at 2026-08-06 20:42:47 on dist-test-slave-tm1g"
server_key: "4e91ba51cadcbb69690baee1fb4c7b14"
server_key_iv: "5c2b78c770d924a22aff41ffba6ef3ee"
server_key_version: "encryptionkey@0"
I20260806 20:42:47.397255 12000 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260806 20:42:47.400662 12016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:47.402012 12000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260806 20:42:47.402295 12000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "9bfca7b285074417a0772d8b2eeaa32a"
format_stamp: "Formatted at 2026-08-06 20:42:47 on dist-test-slave-tm1g"
server_key: "4e91ba51cadcbb69690baee1fb4c7b14"
server_key_iv: "5c2b78c770d924a22aff41ffba6ef3ee"
server_key_version: "encryptionkey@0"
I20260806 20:42:47.402540 12000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:47.435424 12000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:47.439402 12000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:47.439697 12000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:47.458153 12000 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:39471
I20260806 20:42:47.458189 12067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:39471 every 8 connection(s)
I20260806 20:42:47.459491 12000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:47.463063 12068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:47.464329 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12000
I20260806 20:42:47.464524 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:47.464982 24352 external_mini_cluster.cc:1428] Setting key 64bb907be0f69143432184cbd166513e
I20260806 20:42:47.477159 12068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a: Bootstrap starting.
Aug 06 20:42:47 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:47.480708 12068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:47.481904 12068 log.cc:824] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:47.484757 12068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a: No bootstrap required, opened a new log
I20260806 20:42:47.487630 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:47.466794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50880 (local address 127.23.200.62:39471)
0806 20:42:47.467572 (+ 778us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:47.467586 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:47.467625 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:42:47.469213 (+ 1588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:47.469234 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:47.469598 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:47.469921 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:47.471802 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:47.473036 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:47.474989 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:47.475604 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:47.475726 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:47.480516 (+ 4790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:47.480528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:47.480546 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:47.480588 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:47.483668 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:47.484361 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:47.484367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:47.484372 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:47.484535 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:47.484919 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:47.484931 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:47.484933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:47.485456 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:47.485754 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:47.486268 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 20:42:47.486686 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":373,"threads_started":1}
I20260806 20:42:47.492486 12068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfca7b285074417a0772d8b2eeaa32a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39471 } }
I20260806 20:42:47.492851 12068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:47.492960 12068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfca7b285074417a0772d8b2eeaa32a, State: Initialized, Role: FOLLOWER
I20260806 20:42:47.493520 12068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfca7b285074417a0772d8b2eeaa32a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39471 } }
I20260806 20:42:47.493767 12068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:47.493920 12068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:47.494087 12068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:47.496601 12068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfca7b285074417a0772d8b2eeaa32a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39471 } }
I20260806 20:42:47.497081 12068 leader_election.cc:302] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bfca7b285074417a0772d8b2eeaa32a; no voters:
I20260806 20:42:47.497666 12068 leader_election.cc:288] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:47.497754 12073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:47.498539 12073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [term 1 LEADER]: Becoming Leader. State: Replica: 9bfca7b285074417a0772d8b2eeaa32a, State: Running, Role: LEADER
I20260806 20:42:47.499154 12073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfca7b285074417a0772d8b2eeaa32a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39471 } }
I20260806 20:42:47.499912 12068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:47.501957 12075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bfca7b285074417a0772d8b2eeaa32a. Latest consensus state: current_term: 1 leader_uuid: "9bfca7b285074417a0772d8b2eeaa32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfca7b285074417a0772d8b2eeaa32a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39471 } } }
I20260806 20:42:47.501940 12074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bfca7b285074417a0772d8b2eeaa32a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bfca7b285074417a0772d8b2eeaa32a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 39471 } } }
I20260806 20:42:47.502421 12074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:47.502421 12075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:47.504124 12081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:47.509287 12081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:47.517894 12081 catalog_manager.cc:1347] Generated new cluster ID: 2592e91c6ebc498db9a12179db0a1296
I20260806 20:42:47.517997 12081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:47.553987 12081 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:47.555433 12081 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:47.564952 12081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a: Generated new TSK 0
I20260806 20:42:47.565944 12081 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.200.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
Entry for principal kudu/127.23.200.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab.
I20260806 20:42:47.628134 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39471
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:47.966439 12095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:47.966740 12095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:47.966909 12095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:47.974077 12095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:47.974226 12095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:47.974417 12095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:47.983837 12095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.1.keytab
--principal=kudu/127.23.200.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:47.985059 12095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:47.986403 12095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:47.999399 12100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:47.999433 12101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:47.999405 12103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:48.000185 12095 server_base.cc:1034] running on GCE node
I20260806 20:42:48.000695 12095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:48.001674 12095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:48.002904 12095 hybrid_clock.cc:648] HybridClock initialized: now 1786048968002849 us; error 42 us; skew 500 ppm
I20260806 20:42:48.003171 12095 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:48 dist-test-slave-tm1g krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048968, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:48.006464 12095 init.cc:377] Logged in from keytab as kudu/127.23.200.1@KRBTEST.COM (short username kudu)
I20260806 20:42:48.007921 12095 webserver.cc:460] Webserver started at http://127.23.200.1:34443/ using document root <none> and password file <none>
I20260806 20:42:48.008543 12095 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:48.008641 12095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:48.008880 12095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:48.011643 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "522c85467a9e4d7bb0c0ff0a786a9be2"
format_stamp: "Formatted at 2026-08-06 20:42:48 on dist-test-slave-tm1g"
server_key: "02342bfcbc4bd96ffd4491b9cb69c895"
server_key_iv: "ff7a2cc7d52f7bcaab79bb73449a97b8"
server_key_version: "encryptionkey@0"
I20260806 20:42:48.012384 12095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "522c85467a9e4d7bb0c0ff0a786a9be2"
format_stamp: "Formatted at 2026-08-06 20:42:48 on dist-test-slave-tm1g"
server_key: "02342bfcbc4bd96ffd4491b9cb69c895"
server_key_iv: "ff7a2cc7d52f7bcaab79bb73449a97b8"
server_key_version: "encryptionkey@0"
I20260806 20:42:48.017251 12095 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260806 20:42:48.020629 12111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:48.021863 12095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:42:48.022033 12095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "522c85467a9e4d7bb0c0ff0a786a9be2"
format_stamp: "Formatted at 2026-08-06 20:42:48 on dist-test-slave-tm1g"
server_key: "02342bfcbc4bd96ffd4491b9cb69c895"
server_key_iv: "ff7a2cc7d52f7bcaab79bb73449a97b8"
server_key_version: "encryptionkey@0"
I20260806 20:42:48.022225 12095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:48.040825 12095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:48.044855 12095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:48.045132 12095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:48.046047 12095 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:48.047435 12095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:48.047538 12095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:48.047693 12095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:48.047766 12095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:48.081434 12095 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:39765
I20260806 20:42:48.081470 12224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:39765 every 8 connection(s)
I20260806 20:42:48.082799 12095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:48.084874 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12095
I20260806 20:42:48.085072 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:42:48.085574 24352 external_mini_cluster.cc:1428] Setting key 281e01d69661f345d76ebb93e143e2bf
Aug 06 20:42:48 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048968, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.1@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:48.104768 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:48.085483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:49105 (local address 127.23.200.62:39471)
0806 20:42:48.086011 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:48.086017 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:48.087000 (+ 983us) server_negotiation.cc:407] Connection header received
0806 20:42:48.089050 (+ 2050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:48.089056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:48.089171 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:48.089307 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:48.091247 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.091981 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.093497 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.093904 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.094040 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:48.097629 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:48.097639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:48.097647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:48.097692 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:48.101344 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:48.102029 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:48.102038 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:48.102045 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:48.102193 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:48.102694 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:48.102703 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:48.102709 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:48.103089 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:48.103242 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:48.104001 (+ 759us) server_negotiation.cc:299] Negotiation successful
0806 20:42:48.104288 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":272,"threads_started":1}
I20260806 20:42:48.105463 12228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:48.086014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:39471 (local address 127.23.200.1:49105)
0806 20:42:48.086818 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:48.086867 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:48.088785 (+ 1918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:48.089715 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:48.089730 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:48.090529 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:48.091080 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:48.091097 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.092133 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.092138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:48.093354 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:48.093366 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.094147 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.094155 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:48.094470 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:48.095148 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:48.095174 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:48.097416 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:48.101481 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:48.101488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:48.101498 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:48.101901 (+ 403us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:48.102297 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:48.102332 (+ 35us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:48.102339 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:48.102539 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:48.103209 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:48.103214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:48.103503 (+ 289us) client_negotiation.cc:769] Sending connection context
0806 20:42:48.103775 (+ 272us) client_negotiation.cc:240] Negotiation successful
0806 20:42:48.104222 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":336,"threads_started":1}
I20260806 20:42:48.107134 12225 heartbeater.cc:347] Connected to a master server at 127.23.200.62:39471
I20260806 20:42:48.107458 12225 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:48.108310 12225 heartbeater.cc:511] Master 127.23.200.62:39471 requested a full tablet report, sending...
I20260806 20:42:48.110697 12033 ts_manager.cc:194] Registered new tserver with Master: 522c85467a9e4d7bb0c0ff0a786a9be2 (127.23.200.1:39765)
I20260806 20:42:48.112210 12033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.1@KRBTEST.COM'} at 127.23.200.1:49105
WARNING: no policy specified for kudu/127.23.200.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
Entry for principal kudu/127.23.200.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab.
I20260806 20:42:48.144683 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39471
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:48.477154 12231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:48.477486 12231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:48.477632 12231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:48.486145 12231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:48.486249 12231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:48.486433 12231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:48.495818 12231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.2.keytab
--principal=kudu/127.23.200.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:48.496968 12231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:48.498191 12231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:48.511296 12237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:48.511258 12239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:48.511332 12236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:48.512007 12231 server_base.cc:1034] running on GCE node
I20260806 20:42:48.512588 12231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:48.513443 12231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:48.514668 12231 hybrid_clock.cc:648] HybridClock initialized: now 1786048968514630 us; error 53 us; skew 500 ppm
I20260806 20:42:48.514966 12231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:48 dist-test-slave-tm1g krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048968, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:48.518018 12231 init.cc:377] Logged in from keytab as kudu/127.23.200.2@KRBTEST.COM (short username kudu)
I20260806 20:42:48.519505 12231 webserver.cc:460] Webserver started at http://127.23.200.2:33571/ using document root <none> and password file <none>
I20260806 20:42:48.520157 12231 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:48.520300 12231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:48.520623 12231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:48.523557 12231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "e6f37cf7a7574188bd6f7ab99cd92b16"
format_stamp: "Formatted at 2026-08-06 20:42:48 on dist-test-slave-tm1g"
server_key: "5a80c36ea788555efbf0238086a41b0b"
server_key_iv: "eb41bc024f94a5f6ebee544e0bb0f195"
server_key_version: "encryptionkey@0"
I20260806 20:42:48.524379 12231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "e6f37cf7a7574188bd6f7ab99cd92b16"
format_stamp: "Formatted at 2026-08-06 20:42:48 on dist-test-slave-tm1g"
server_key: "5a80c36ea788555efbf0238086a41b0b"
server_key_iv: "eb41bc024f94a5f6ebee544e0bb0f195"
server_key_version: "encryptionkey@0"
I20260806 20:42:48.529546 12231 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260806 20:42:48.532795 12246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:48.534042 12231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:42:48.534216 12231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "e6f37cf7a7574188bd6f7ab99cd92b16"
format_stamp: "Formatted at 2026-08-06 20:42:48 on dist-test-slave-tm1g"
server_key: "5a80c36ea788555efbf0238086a41b0b"
server_key_iv: "eb41bc024f94a5f6ebee544e0bb0f195"
server_key_version: "encryptionkey@0"
I20260806 20:42:48.534493 12231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:48.546767 12231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:48.550853 12231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:48.551137 12231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:48.552044 12231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:48.553284 12231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:48.553388 12231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:48.553534 12231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:48.553607 12231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:48.586867 12231 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:38367
I20260806 20:42:48.586920 12359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:38367 every 8 connection(s)
I20260806 20:42:48.588122 12231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:48.597291 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12231
I20260806 20:42:48.597496 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:42:48.597937 24352 external_mini_cluster.cc:1428] Setting key 70aae9448da27f74d1da09aaac8e3121
Aug 06 20:42:48 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048968, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.2@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:48.607703 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:48.590559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:35631 (local address 127.23.200.62:39471)
0806 20:42:48.590742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:48.590746 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:48.591839 (+ 1093us) server_negotiation.cc:407] Connection header received
0806 20:42:48.593564 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:48.593568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:48.593636 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:48.593723 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:48.595167 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.595872 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.597339 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.597862 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.598004 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:48.601851 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:48.601856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:48.601860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:48.601898 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:48.604517 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:48.605307 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:48.605311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:48.605313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:48.605442 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:48.605868 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:48.605879 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:48.605887 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:48.606254 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:48.606464 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:48.607109 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 20:42:48.607337 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 20:42:48.608637 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:48.590964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:39471 (local address 127.23.200.2:35631)
0806 20:42:48.591669 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:48.591708 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:48.593311 (+ 1603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:48.593915 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:48.593922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:48.594485 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:48.595012 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:48.595029 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.595996 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.595999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:48.597179 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:48.597190 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:48.598087 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:48.598094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:48.598375 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:48.599183 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:48.599209 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:48.601639 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:48.604651 (+ 3012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:48.604660 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:48.604674 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:48.605169 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:48.605558 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:48.605563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:48.605566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:48.605745 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:48.606429 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:48.606436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:48.606751 (+ 315us) client_negotiation.cc:769] Sending connection context
0806 20:42:48.607002 (+ 251us) client_negotiation.cc:240] Negotiation successful
0806 20:42:48.607544 (+ 542us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":281,"threads_started":1}
I20260806 20:42:48.610021 12360 heartbeater.cc:347] Connected to a master server at 127.23.200.62:39471
I20260806 20:42:48.610365 12360 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:48.611182 12360 heartbeater.cc:511] Master 127.23.200.62:39471 requested a full tablet report, sending...
I20260806 20:42:48.612702 12033 ts_manager.cc:194] Registered new tserver with Master: e6f37cf7a7574188bd6f7ab99cd92b16 (127.23.200.2:38367)
I20260806 20:42:48.613946 12033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.2@KRBTEST.COM'} at 127.23.200.2:35631
WARNING: no policy specified for kudu/127.23.200.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
Entry for principal kudu/127.23.200.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab.
I20260806 20:42:48.654734 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39471
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:48.979295 12365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:48.979552 12365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:48.979728 12365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:48.987362 12365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:48.987485 12365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:48.987676 12365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:48.996996 12365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.3.keytab
--principal=kudu/127.23.200.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:39471
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:48.998183 12365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:48.999450 12365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:49.012410 12370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:49.012610 12371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:49.012800 12373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:49.013567 12365 server_base.cc:1034] running on GCE node
I20260806 20:42:49.014164 12365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:49.015075 12365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:49.016305 12365 hybrid_clock.cc:648] HybridClock initialized: now 1786048969016261 us; error 54 us; skew 500 ppm
I20260806 20:42:49.016613 12365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:49 dist-test-slave-tm1g krb5kdc[11987](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048969, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:49.019685 12365 init.cc:377] Logged in from keytab as kudu/127.23.200.3@KRBTEST.COM (short username kudu)
I20260806 20:42:49.021090 12365 webserver.cc:460] Webserver started at http://127.23.200.3:39127/ using document root <none> and password file <none>
I20260806 20:42:49.021698 12365 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:49.021807 12365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:49.022081 12365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:49.024924 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "becdaec17a7748349040154edcf23ca5"
format_stamp: "Formatted at 2026-08-06 20:42:49 on dist-test-slave-tm1g"
server_key: "512d6c6c709bb75499dd898133b14cf5"
server_key_iv: "40ce4f54c0e0499fe501294ca0be8c39"
server_key_version: "encryptionkey@0"
I20260806 20:42:49.025687 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "becdaec17a7748349040154edcf23ca5"
format_stamp: "Formatted at 2026-08-06 20:42:49 on dist-test-slave-tm1g"
server_key: "512d6c6c709bb75499dd898133b14cf5"
server_key_iv: "40ce4f54c0e0499fe501294ca0be8c39"
server_key_version: "encryptionkey@0"
I20260806 20:42:49.030740 12365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260806 20:42:49.034025 12380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:49.035542 12365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:49.035706 12365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "becdaec17a7748349040154edcf23ca5"
format_stamp: "Formatted at 2026-08-06 20:42:49 on dist-test-slave-tm1g"
server_key: "512d6c6c709bb75499dd898133b14cf5"
server_key_iv: "40ce4f54c0e0499fe501294ca0be8c39"
server_key_version: "encryptionkey@0"
I20260806 20:42:49.035884 12365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:49.060499 12365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:49.064792 12365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:49.065094 12365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:49.066123 12365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:49.067412 12365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:49.067487 12365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:49.067662 12365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:49.067713 12365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:49.100937 12365 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:39941
I20260806 20:42:49.100967 12493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:39941 every 8 connection(s)
I20260806 20:42:49.102193 12365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:49.106436 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12365
I20260806 20:42:49.106630 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:42:49.107088 24352 external_mini_cluster.cc:1428] Setting key 7b0746465ab19d7eb3f7a3ab199b66df
I20260806 20:42:49.115581 12225 heartbeater.cc:503] Master 127.23.200.62:39471 was elected leader, sending a full tablet report...
Aug 06 20:42:49 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048969, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.3@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:42:49.122658 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.104727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:59639 (local address 127.23.200.62:39471)
0806 20:42:49.104954 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.104958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.106178 (+ 1220us) server_negotiation.cc:407] Connection header received
0806 20:42:49.108624 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.108629 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.108726 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.108832 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:49.110380 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.111165 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.112664 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.113097 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.113227 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.116915 (+ 3688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:49.116920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:49.116923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:49.116972 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:49.119433 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.120216 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.120221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.120225 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.120359 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.120862 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.120870 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.120874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.121176 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:49.121326 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.122169 (+ 843us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.122413 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260806 20:42:49.123529 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.105235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:39471 (local address 127.23.200.3:59639)
0806 20:42:49.106021 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:49.106069 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:49.108314 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:49.109064 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:49.109082 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:49.109566 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:49.110201 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.110217 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.111318 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.111323 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.112521 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.112538 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.113324 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.113331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.113581 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.114276 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:42:49.114297 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:49.116721 (+ 2424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:49.119552 (+ 2831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:49.119558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:49.119568 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:49.119982 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:49.120475 (+ 493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:42:49.120481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:42:49.120484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:42:49.120690 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:42:49.121324 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:49.121335 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:49.121714 (+ 379us) client_negotiation.cc:769] Sending connection context
0806 20:42:49.122023 (+ 309us) client_negotiation.cc:240] Negotiation successful
0806 20:42:49.122487 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":278,"threads_started":1}
I20260806 20:42:49.124775 12494 heartbeater.cc:347] Connected to a master server at 127.23.200.62:39471
I20260806 20:42:49.125033 12494 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:49.125787 12494 heartbeater.cc:511] Master 127.23.200.62:39471 requested a full tablet report, sending...
I20260806 20:42:49.127171 12033 ts_manager.cc:194] Registered new tserver with Master: becdaec17a7748349040154edcf23ca5 (127.23.200.3:39941)
I20260806 20:42:49.127954 12033 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.200.3@KRBTEST.COM'} at 127.23.200.3:59639
I20260806 20:42:49.133946 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:42:49.150506 12227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.138019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.23.200.62:39471)
0806 20:42:49.138224 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.138231 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.138678 (+ 447us) server_negotiation.cc:407] Connection header received
0806 20:42:49.138911 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.138916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.139010 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.139110 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:49.140583 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.141292 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.142611 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.143049 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.143182 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.145500 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:49.145506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:49.145511 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:49.145551 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:49.147909 (+ 2358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.148553 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.148560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.148563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.148695 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.149054 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.149062 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.149066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.149361 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:49.149621 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.149947 (+ 326us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.150163 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260806 20:42:49.153818 12033 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52768:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:42:49.160542 12033 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:42:49.193094 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.172128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36236 (local address 127.23.200.2:38367)
0806 20:42:49.184641 (+ 12513us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.184649 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.184672 (+ 23us) server_negotiation.cc:407] Connection header received
0806 20:42:49.184801 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.184810 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.185093 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.185340 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:49.186398 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.188041 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.190404 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.191874 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.192031 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.192242 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.192548 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.192730 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12345,"thread_start_us":12327,"threads_started":1}
I20260806 20:42:49.193981 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.172521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34942 (local address 127.23.200.3:39941)
0806 20:42:49.184375 (+ 11854us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.184385 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.184449 (+ 64us) server_negotiation.cc:407] Connection header received
0806 20:42:49.184565 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.184573 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.184843 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.185057 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:49.186222 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.187915 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.190707 (+ 2792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.192050 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.192160 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.192349 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.193485 (+ 1136us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.193658 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11675,"thread_start_us":11491,"threads_started":1}
I20260806 20:42:49.194069 12507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.171943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:38367 (local address 127.0.0.1:36236)
0806 20:42:49.183510 (+ 11567us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:49.183575 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:49.183800 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:49.185567 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:49.185573 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:49.185603 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:49.186152 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.186165 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.188159 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.188165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.190228 (+ 2063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.190245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.192090 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.192097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.192280 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.192340 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 20:42:49.192483 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 20:42:49.192675 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11232,"thread_start_us":11995,"threads_started":1}
I20260806 20:42:49.194115 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.184764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36818 (local address 127.23.200.1:39765)
0806 20:42:49.185494 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.185503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.186319 (+ 816us) server_negotiation.cc:407] Connection header received
0806 20:42:49.186622 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.186633 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.186855 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.186997 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:49.187674 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.189240 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.191426 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.192611 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.192729 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.192968 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.193634 (+ 666us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.193797 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":412,"threads_started":1}
I20260806 20:42:49.194108 12511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.185479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39765 (local address 127.0.0.1:36818)
0806 20:42:49.186185 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:49.186205 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:49.186418 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:49.187218 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:49.187221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:49.187240 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:49.187509 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.187516 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.189433 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.189439 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.191276 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.191287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.193284 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.193290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.193457 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.193495 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 20:42:49.193591 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 20:42:49.193685 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":48,"thread_start_us":372,"threads_started":1}
I20260806 20:42:49.195070 12506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.172149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:39941 (local address 127.0.0.1:34942)
0806 20:42:49.184240 (+ 12091us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:49.184261 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:49.184370 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:49.185434 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:49.185441 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:49.185472 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:49.185964 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.185981 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.188060 (+ 2079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.188066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.190537 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.190553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.193004 (+ 2451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.193013 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.193209 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.193255 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 20:42:49.194573 (+ 1318us) client_negotiation.cc:240] Negotiation successful
0806 20:42:49.194697 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11949,"mutex_wait_us":106,"thread_start_us":10843,"threads_started":1}
I20260806 20:42:49.198177 12295 tablet_service.cc:1467] Processing CreateTablet for tablet 89af858a185a41768a48e2ec2938271b (DEFAULT_TABLE table=test-table [id=d363866ab3f6485fb7ee0ae7ed14fc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:49.198462 12159 tablet_service.cc:1467] Processing CreateTablet for tablet 89af858a185a41768a48e2ec2938271b (DEFAULT_TABLE table=test-table [id=d363866ab3f6485fb7ee0ae7ed14fc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:49.199746 12295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89af858a185a41768a48e2ec2938271b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:49.200018 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89af858a185a41768a48e2ec2938271b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:49.200898 12429 tablet_service.cc:1467] Processing CreateTablet for tablet 89af858a185a41768a48e2ec2938271b (DEFAULT_TABLE table=test-table [id=d363866ab3f6485fb7ee0ae7ed14fc28]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:49.202253 12429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89af858a185a41768a48e2ec2938271b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:49.211549 12512 tablet_bootstrap.cc:492] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Bootstrap starting.
I20260806 20:42:49.214584 12513 tablet_bootstrap.cc:492] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Bootstrap starting.
I20260806 20:42:49.215616 12512 tablet_bootstrap.cc:654] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:49.216820 12514 tablet_bootstrap.cc:492] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Bootstrap starting.
I20260806 20:42:49.216878 12512 log.cc:824] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:49.218060 12513 tablet_bootstrap.cc:654] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:49.219228 12513 log.cc:824] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:49.220009 12514 tablet_bootstrap.cc:654] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:49.221148 12514 log.cc:824] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:49.221271 12512 tablet_bootstrap.cc:492] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: No bootstrap required, opened a new log
I20260806 20:42:49.221585 12512 ts_tablet_manager.cc:1397] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 20:42:49.223778 12514 tablet_bootstrap.cc:492] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: No bootstrap required, opened a new log
I20260806 20:42:49.224031 12514 ts_tablet_manager.cc:1397] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 20:42:49.224677 12513 tablet_bootstrap.cc:492] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: No bootstrap required, opened a new log
I20260806 20:42:49.224908 12513 ts_tablet_manager.cc:1397] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260806 20:42:49.231520 12514 raft_consensus.cc:348] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.231906 12514 raft_consensus.cc:374] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:49.232048 12514 raft_consensus.cc:729] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: becdaec17a7748349040154edcf23ca5, State: Initialized, Role: FOLLOWER
I20260806 20:42:49.232353 12512 raft_consensus.cc:348] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.232761 12512 raft_consensus.cc:374] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:49.232648 12514 consensus_queue.cc:260] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.232854 12512 raft_consensus.cc:729] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6f37cf7a7574188bd6f7ab99cd92b16, State: Initialized, Role: FOLLOWER
I20260806 20:42:49.233393 12512 consensus_queue.cc:260] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.234694 12360 heartbeater.cc:503] Master 127.23.200.62:39471 was elected leader, sending a full tablet report...
I20260806 20:42:49.235240 12512 ts_tablet_manager.cc:1428] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Time spent starting tablet: real 0.013s user 0.007s sys 0.005s
I20260806 20:42:49.235561 12513 raft_consensus.cc:348] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.235929 12513 raft_consensus.cc:374] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:49.236027 12513 raft_consensus.cc:729] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 522c85467a9e4d7bb0c0ff0a786a9be2, State: Initialized, Role: FOLLOWER
I20260806 20:42:49.236652 12513 consensus_queue.cc:260] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.238750 12513 ts_tablet_manager.cc:1428] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260806 20:42:49.240593 12514 ts_tablet_manager.cc:1428] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Time spent starting tablet: real 0.016s user 0.007s sys 0.010s
I20260806 20:42:49.240943 12494 heartbeater.cc:503] Master 127.23.200.62:39471 was elected leader, sending a full tablet report...
W20260806 20:42:49.335188 12226 tablet.cc:2369] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:42:49.340044 12361 tablet.cc:2369] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:42:49.353746 12495 tablet.cc:2369] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:49.368425 12518 raft_consensus.cc:482] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:49.368677 12518 raft_consensus.cc:504] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.370275 12518 leader_election.cc:288] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6f37cf7a7574188bd6f7ab99cd92b16 (127.23.200.2:38367), 522c85467a9e4d7bb0c0ff0a786a9be2 (127.23.200.1:39765)
I20260806 20:42:49.376819 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.370584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:38367 (local address 127.23.200.3:55399)
0806 20:42:49.370786 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:49.370807 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:49.370927 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:49.371512 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:49.371517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:49.371548 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:49.371765 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.371777 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.373024 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.373027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.374993 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.375008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.376302 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.376306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.376437 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.376461 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 20:42:49.376532 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 20:42:49.376622 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":57}
I20260806 20:42:49.376983 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.370729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:55399 (local address 127.23.200.2:38367)
0806 20:42:49.370965 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.370972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.371022 (+ 50us) server_negotiation.cc:407] Connection header received
0806 20:42:49.371104 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.371109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.371201 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.371311 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:49.371914 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.372901 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.375151 (+ 2250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.376169 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.376321 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.376399 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.376590 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.376722 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 20:42:49.377835 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.371408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:52877 (local address 127.23.200.1:39765)
0806 20:42:49.371597 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.371601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.371745 (+ 144us) server_negotiation.cc:407] Connection header received
0806 20:42:49.371917 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.371921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.372000 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.372098 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:49.372590 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.373434 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.375631 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.376810 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.376955 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.377033 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.377468 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.377595 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:42:49.377940 12522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.370970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:39765 (local address 127.23.200.3:52877)
0806 20:42:49.371643 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:49.371659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:49.371779 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:49.372115 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:49.372117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:49.372135 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:49.372437 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.372445 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.373582 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.373587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.375480 (+ 1893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:49.375491 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.377173 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.377180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:49.377310 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.377331 (+ 21us) client_negotiation.cc:769] Sending connection context
0806 20:42:49.377438 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 20:42:49.377568 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"mutex_wait_us":1,"thread_start_us":318,"threads_started":1}
I20260806 20:42:49.377846 12315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89af858a185a41768a48e2ec2938271b" candidate_uuid: "becdaec17a7748349040154edcf23ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" is_pre_election: true
I20260806 20:42:49.378495 12315 raft_consensus.cc:2454] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate becdaec17a7748349040154edcf23ca5 in term 0.
I20260806 20:42:49.379129 12180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89af858a185a41768a48e2ec2938271b" candidate_uuid: "becdaec17a7748349040154edcf23ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" is_pre_election: true
I20260806 20:42:49.379254 12381 leader_election.cc:302] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: becdaec17a7748349040154edcf23ca5, e6f37cf7a7574188bd6f7ab99cd92b16; no voters:
I20260806 20:42:49.379689 12518 raft_consensus.cc:2781] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:42:49.379666 12180 raft_consensus.cc:2454] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate becdaec17a7748349040154edcf23ca5 in term 0.
I20260806 20:42:49.379859 12518 raft_consensus.cc:482] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:42:49.379982 12518 raft_consensus.cc:3037] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:49.382383 12518 raft_consensus.cc:504] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.382899 12518 leader_election.cc:288] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [CANDIDATE]: Term 1 election: Requested vote from peers e6f37cf7a7574188bd6f7ab99cd92b16 (127.23.200.2:38367), 522c85467a9e4d7bb0c0ff0a786a9be2 (127.23.200.1:39765)
I20260806 20:42:49.383325 12315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89af858a185a41768a48e2ec2938271b" candidate_uuid: "becdaec17a7748349040154edcf23ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16"
I20260806 20:42:49.383381 12180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89af858a185a41768a48e2ec2938271b" candidate_uuid: "becdaec17a7748349040154edcf23ca5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2"
I20260806 20:42:49.383518 12315 raft_consensus.cc:3037] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:49.383586 12180 raft_consensus.cc:3037] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:49.385682 12315 raft_consensus.cc:2454] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate becdaec17a7748349040154edcf23ca5 in term 1.
I20260806 20:42:49.385746 12180 raft_consensus.cc:2454] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate becdaec17a7748349040154edcf23ca5 in term 1.
I20260806 20:42:49.386065 12381 leader_election.cc:302] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: becdaec17a7748349040154edcf23ca5, e6f37cf7a7574188bd6f7ab99cd92b16; no voters:
I20260806 20:42:49.386415 12518 raft_consensus.cc:2781] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:49.386854 12518 raft_consensus.cc:686] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 1 LEADER]: Becoming Leader. State: Replica: becdaec17a7748349040154edcf23ca5, State: Running, Role: LEADER
I20260806 20:42:49.387363 12518 consensus_queue.cc:237] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } }
I20260806 20:42:49.391784 12032 catalog_manager.cc:5495] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 reported cstate change: term changed from 0 to 1, leader changed from <none> to becdaec17a7748349040154edcf23ca5 (127.23.200.3). New cstate: current_term: 1 leader_uuid: "becdaec17a7748349040154edcf23ca5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "becdaec17a7748349040154edcf23ca5" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 39941 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 } health_report { overall_health: UNKNOWN } } }
I20260806 20:42:49.436904 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.431913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34958 (local address 127.23.200.3:39941)
0806 20:42:49.432091 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.432095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.432278 (+ 183us) server_negotiation.cc:407] Connection header received
0806 20:42:49.432448 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.432452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.432536 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.432686 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:42:49.433004 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.433713 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.435020 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.435414 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.435508 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.435958 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:42:49.436443 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:42:49.436555 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.436648 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.436711 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 20:42:49.445055 12315 raft_consensus.cc:1261] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 1 FOLLOWER]: Refusing update from remote peer becdaec17a7748349040154edcf23ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:49.445055 12180 raft_consensus.cc:1261] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 1 FOLLOWER]: Refusing update from remote peer becdaec17a7748349040154edcf23ca5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:49.446019 12523 consensus_queue.cc:1035] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "522c85467a9e4d7bb0c0ff0a786a9be2" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:49.446409 12518 consensus_queue.cc:1035] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6f37cf7a7574188bd6f7ab99cd92b16" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 38367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:49.493697 12033 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:49.493990 12033 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52768:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:49.497627 12033 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bfca7b285074417a0772d8b2eeaa32a: Sending DeleteTablet for 3 replicas of tablet 89af858a185a41768a48e2ec2938271b
I20260806 20:42:49.498750 12429 tablet_service.cc:1514] Processing DeleteTablet for tablet 89af858a185a41768a48e2ec2938271b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:49 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:34942
I20260806 20:42:49.499135 12295 tablet_service.cc:1514] Processing DeleteTablet for tablet 89af858a185a41768a48e2ec2938271b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:49 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:36236
I20260806 20:42:49.499279 12159 tablet_service.cc:1514] Processing DeleteTablet for tablet 89af858a185a41768a48e2ec2938271b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:49 UTC) from {username='kudu', principal='kudu/127.23.200.62@KRBTEST.COM'} at 127.0.0.1:36818
I20260806 20:42:49.500728 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12095
I20260806 20:42:49.509689 12538 tablet_replica.cc:330] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: stopping tablet replica
I20260806 20:42:49.510196 12538 raft_consensus.cc:2229] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:49.510761 12538 raft_consensus.cc:2258] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:49.513157 12538 ts_tablet_manager.cc:1905] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:49.515486 12537 tablet_replica.cc:330] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: stopping tablet replica
I20260806 20:42:49.515971 12537 raft_consensus.cc:2229] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:42:49.516649 12537 raft_consensus.cc:2258] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:49.518451 12537 ts_tablet_manager.cc:1905] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:49.521819 12538 ts_tablet_manager.cc:1918] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:49.522002 12538 log.cc:1196] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/89af858a185a41768a48e2ec2938271b
I20260806 20:42:49.522619 12538 ts_tablet_manager.cc:1939] T 89af858a185a41768a48e2ec2938271b P e6f37cf7a7574188bd6f7ab99cd92b16: Deleting consensus metadata
I20260806 20:42:49.524587 12017 catalog_manager.cc:4841] TS e6f37cf7a7574188bd6f7ab99cd92b16 (127.23.200.2:38367): tablet 89af858a185a41768a48e2ec2938271b (table test-table [id=d363866ab3f6485fb7ee0ae7ed14fc28]) successfully deleted
I20260806 20:42:49.526209 12537 ts_tablet_manager.cc:1918] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:49.526587 12537 log.cc:1196] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/89af858a185a41768a48e2ec2938271b
I20260806 20:42:49.526608 12539 tablet_replica.cc:330] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: stopping tablet replica
I20260806 20:42:49.526968 12539 raft_consensus.cc:2229] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:49.527135 12537 ts_tablet_manager.cc:1939] T 89af858a185a41768a48e2ec2938271b P becdaec17a7748349040154edcf23ca5: Deleting consensus metadata
I20260806 20:42:49.527451 12539 raft_consensus.cc:2258] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:49.528759 12019 catalog_manager.cc:4841] TS becdaec17a7748349040154edcf23ca5 (127.23.200.3:39941): tablet 89af858a185a41768a48e2ec2938271b (table test-table [id=d363866ab3f6485fb7ee0ae7ed14fc28]) successfully deleted
I20260806 20:42:49.531715 12539 ts_tablet_manager.cc:1905] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 20:42:49 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.1@KRBTEST.COM
I20260806 20:42:49.537904 12539 ts_tablet_manager.cc:1918] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:49.538074 12539 log.cc:1196] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/89af858a185a41768a48e2ec2938271b
I20260806 20:42:49.538821 12539 ts_tablet_manager.cc:1939] T 89af858a185a41768a48e2ec2938271b P 522c85467a9e4d7bb0c0ff0a786a9be2: Deleting consensus metadata
I20260806 20:42:49.540712 12017 catalog_manager.cc:4841] TS 522c85467a9e4d7bb0c0ff0a786a9be2 (127.23.200.1:39765): tablet 89af858a185a41768a48e2ec2938271b (table test-table [id=d363866ab3f6485fb7ee0ae7ed14fc28]) successfully deleted
I20260806 20:42:49.544430 12510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.501918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36822 (local address 127.23.200.1:39765)
0806 20:42:49.502097 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.502104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.525736 (+ 23632us) server_negotiation.cc:407] Connection header received
0806 20:42:49.526041 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.526048 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.526167 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.526331 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:49.529232 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.530013 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.531897 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.532591 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.532731 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.538025 (+ 5294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:49.538039 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:49.538058 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:49.538103 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:49.541670 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.542538 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.542544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.542546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.542667 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.543080 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.543083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.543085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.543421 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:49.543633 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.543978 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.544127 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:42:49.545128 12220 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:49.648789 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12095
I20260806 20:42:49.674546 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12231
Aug 06 20:42:49 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.2@KRBTEST.COM
I20260806 20:42:49.692044 12508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.675470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.23.200.2:38367)
0806 20:42:49.675649 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.675653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.675671 (+ 18us) server_negotiation.cc:407] Connection header received
0806 20:42:49.675794 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.675798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.675882 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.675966 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:49.677669 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.678522 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.680343 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.680737 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.680824 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.686285 (+ 5461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:49.686335 (+ 50us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:49.686361 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:49.686400 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:49.689161 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.689960 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.689967 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.689969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.690086 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.690667 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.690671 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.690672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.691027 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:49.691245 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.691583 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.691751 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:42:49.692823 12355 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:49.786881 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12231
I20260806 20:42:49.811439 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12365
Aug 06 20:42:49 dist-test-slave-tm1g krb5kdc[11987](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048966, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.3@KRBTEST.COM
I20260806 20:42:49.829648 12509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:49.812317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.23.200.3:39941)
0806 20:42:49.812507 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:49.812511 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:49.812572 (+ 61us) server_negotiation.cc:407] Connection header received
0806 20:42:49.812766 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:49.812770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:49.812834 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:49.812928 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:49.814812 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.815578 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.817571 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:49.817986 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:49.818081 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:49.823520 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:49.823533 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:49.823550 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:49.823590 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:49.826776 (+ 3186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.827478 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.827485 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.827488 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.827640 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:49.828030 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:49.828037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:49.828045 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:49.828410 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:49.828743 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:49.829041 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 20:42:49.829275 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:42:49.830427 12489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:49.952718 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12365
I20260806 20:42:49.973775 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12000
I20260806 20:42:49.974960 12063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:50.084342 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12000
2026-08-06T20:42:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4446 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T20:42:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:50Z Disabled control of system clock
I20260806 20:42:50.161580 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:33191
--webserver_interface=127.23.200.62
--webserver_port=0
--builtin_ntp_servers=127.23.200.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:33191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 20:42:50.488529 12548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:50.488808 12548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:50.488984 12548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:50.495990 12548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:50.496114 12548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:50.496215 12548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:50.496321 12548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:50.496407 12548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:50.505165 12548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:33191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:33191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:50.506454 12548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:50.508237 12548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:50.519992 12553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:50.520063 12556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:50.520466 12554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:50.520710 12548 server_base.cc:1034] running on GCE node
I20260806 20:42:50.521426 12548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:50.522779 12548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:50.524008 12548 hybrid_clock.cc:648] HybridClock initialized: now 1786048970523975 us; error 43 us; skew 500 ppm
I20260806 20:42:50.524339 12548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:50.526526 12548 webserver.cc:460] Webserver started at http://127.23.200.62:38909/ using document root <none> and password file <none>
I20260806 20:42:50.527195 12548 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:50.527324 12548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:50.527681 12548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:50.530508 12548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "df997f5658f94a6bb563b724e2af4dec"
format_stamp: "Formatted at 2026-08-06 20:42:50 on dist-test-slave-tm1g"
server_key: "59972072adcae34e8ace6fccf7c993a5"
server_key_iv: "d4307b8da1d5db0388311328a2900673"
server_key_version: "encryptionkey@0"
I20260806 20:42:50.531286 12548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "df997f5658f94a6bb563b724e2af4dec"
format_stamp: "Formatted at 2026-08-06 20:42:50 on dist-test-slave-tm1g"
server_key: "59972072adcae34e8ace6fccf7c993a5"
server_key_iv: "d4307b8da1d5db0388311328a2900673"
server_key_version: "encryptionkey@0"
I20260806 20:42:50.536190 12548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260806 20:42:50.539454 12562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:50.540727 12548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260806 20:42:50.540918 12548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "df997f5658f94a6bb563b724e2af4dec"
format_stamp: "Formatted at 2026-08-06 20:42:50 on dist-test-slave-tm1g"
server_key: "59972072adcae34e8ace6fccf7c993a5"
server_key_iv: "d4307b8da1d5db0388311328a2900673"
server_key_version: "encryptionkey@0"
I20260806 20:42:50.541126 12548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:50.565506 12548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:50.566370 12548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:50.566655 12548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:50.584700 12548 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:33191
I20260806 20:42:50.584720 12613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:33191 every 8 connection(s)
I20260806 20:42:50.585980 12548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:50.589479 12614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:50.592404 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12548
I20260806 20:42:50.592597 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:50.593155 24352 external_mini_cluster.cc:1428] Setting key 73bd0a5887e0c964a0e445e6dde3b98f
I20260806 20:42:50.600818 12614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec: Bootstrap starting.
I20260806 20:42:50.603732 12614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:50.604755 12614 log.cc:824] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:50.606384 12617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:50.595086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.23.200.62:33191)
0806 20:42:50.596023 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:50.596034 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:50.596071 (+ 37us) server_negotiation.cc:407] Connection header received
0806 20:42:50.597664 (+ 1593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:50.597685 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:50.598120 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:50.598573 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:50.599129 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:50.600365 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:50.601738 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:50.602500 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:50.602691 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:50.603827 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:50.603856 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:50.603874 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:50.604411 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:50.604493 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:50.604516 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:50.604644 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:50.604856 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 20:42:50.605143 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":346,"threads_started":1}
I20260806 20:42:50.607807 12614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec: No bootstrap required, opened a new log
I20260806 20:42:50.615270 12614 raft_consensus.cc:348] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df997f5658f94a6bb563b724e2af4dec" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 33191 } }
I20260806 20:42:50.615617 12614 raft_consensus.cc:374] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:50.615688 12614 raft_consensus.cc:729] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df997f5658f94a6bb563b724e2af4dec, State: Initialized, Role: FOLLOWER
I20260806 20:42:50.616129 12614 consensus_queue.cc:260] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df997f5658f94a6bb563b724e2af4dec" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 33191 } }
I20260806 20:42:50.616317 12614 raft_consensus.cc:388] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:50.616406 12614 raft_consensus.cc:482] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:50.616516 12614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:50.618572 12614 raft_consensus.cc:504] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df997f5658f94a6bb563b724e2af4dec" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 33191 } }
I20260806 20:42:50.619180 12614 leader_election.cc:302] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df997f5658f94a6bb563b724e2af4dec; no voters:
I20260806 20:42:50.619782 12614 leader_election.cc:288] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:50.619958 12619 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:50.620659 12619 raft_consensus.cc:686] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [term 1 LEADER]: Becoming Leader. State: Replica: df997f5658f94a6bb563b724e2af4dec, State: Running, Role: LEADER
I20260806 20:42:50.621142 12619 consensus_queue.cc:237] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df997f5658f94a6bb563b724e2af4dec" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 33191 } }
I20260806 20:42:50.621752 12614 sys_catalog.cc:564] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:50.624259 12620 sys_catalog.cc:455] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df997f5658f94a6bb563b724e2af4dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df997f5658f94a6bb563b724e2af4dec" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 33191 } } }
I20260806 20:42:50.624847 12620 sys_catalog.cc:458] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:50.624234 12621 sys_catalog.cc:455] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader df997f5658f94a6bb563b724e2af4dec. Latest consensus state: current_term: 1 leader_uuid: "df997f5658f94a6bb563b724e2af4dec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df997f5658f94a6bb563b724e2af4dec" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 33191 } } }
I20260806 20:42:50.625196 12621 sys_catalog.cc:458] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:50.626833 12626 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:50.631111 12626 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:50.639331 12626 catalog_manager.cc:1347] Generated new cluster ID: 3b7f255d03d340088d0f25b45ec4b8e2
I20260806 20:42:50.639464 12626 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:50.662974 12626 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:50.664278 12626 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:50.672614 12626 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec: Generated new TSK 0
I20260806 20:42:50.673345 12626 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:42:50.679370 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.1:0
--local_ip_for_outbound_sockets=127.23.200.1
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:33191
--builtin_ntp_servers=127.23.200.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 20:42:51.016201 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:51.016463 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:51.016547 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:51.023666 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:51.023751 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:51.023849 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.1
I20260806 20:42:51.032653 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.200.1
--webserver_port=0
--tserver_master_addrs=127.23.200.62:33191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.1
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:51.033754 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:51.035547 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:51.048460 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:51.048502 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:51.048549 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:51.049415 12638 server_base.cc:1034] running on GCE node
I20260806 20:42:51.049932 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:51.050905 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:51.052145 12638 hybrid_clock.cc:648] HybridClock initialized: now 1786048971052079 us; error 76 us; skew 500 ppm
I20260806 20:42:51.052462 12638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:51.054596 12638 webserver.cc:460] Webserver started at http://127.23.200.1:37225/ using document root <none> and password file <none>
I20260806 20:42:51.055264 12638 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:51.055401 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:51.055727 12638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:51.058499 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data/instance:
uuid: "ec45ea1ed3284ba487e65f8a260ff6e0"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "76baf9d8d8e2841081f40fbc99b44f63"
server_key_iv: "3c35688a98ba92766cab9d3c70ab6460"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.059361 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance:
uuid: "ec45ea1ed3284ba487e65f8a260ff6e0"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "76baf9d8d8e2841081f40fbc99b44f63"
server_key_iv: "3c35688a98ba92766cab9d3c70ab6460"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.064191 12638 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 20:42:51.067441 12652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.068652 12638 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260806 20:42:51.068830 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal
uuid: "ec45ea1ed3284ba487e65f8a260ff6e0"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "76baf9d8d8e2841081f40fbc99b44f63"
server_key_iv: "3c35688a98ba92766cab9d3c70ab6460"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.069042 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:51.079625 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:51.080346 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:51.080654 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:51.081564 12638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:51.083009 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:51.083138 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.083277 12638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:51.083379 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.124083 12638 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.1:43789
I20260806 20:42:51.124109 12766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.1:43789 every 8 connection(s)
I20260806 20:42:51.125658 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/data/info.pb
I20260806 20:42:51.134032 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12638
I20260806 20:42:51.134240 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal/instance
I20260806 20:42:51.134938 24352 external_mini_cluster.cc:1428] Setting key 5c90d3f2f2c8ae3aabde2596b39e6549
I20260806 20:42:51.141024 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.2:0
--local_ip_for_outbound_sockets=127.23.200.2
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:33191
--builtin_ntp_servers=127.23.200.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 20:42:51.141686 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:51.129197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:33191 (local address 127.23.200.1:47411)
0806 20:42:51.130290 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:51.130372 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:51.132132 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:51.132829 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:51.132840 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:51.132958 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:51.133809 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:51.133832 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.135082 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.135089 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:51.136726 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:51.136738 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.137596 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.137606 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:51.137840 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:51.138714 (+ 874us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:51.138716 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:51.138763 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:51.138836 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:51.139724 (+ 888us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:51.139739 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:51.139749 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 20:42:51.139945 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 20:42:51.140311 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":344,"threads_started":1}
I20260806 20:42:51.142647 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:51.128510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.1:47411 (local address 127.23.200.62:33191)
0806 20:42:51.129160 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:51.129167 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:51.130488 (+ 1321us) server_negotiation.cc:407] Connection header received
0806 20:42:51.132401 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:51.132407 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:51.132495 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:51.132619 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:51.134011 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.134922 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.136865 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.137355 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.137502 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:51.139534 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:51.139542 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:51.139549 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:51.139571 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:51.139622 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:51.139638 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:51.139724 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:51.142079 (+ 2355us) server_negotiation.cc:299] Negotiation successful
0806 20:42:51.142191 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":352,"threads_started":1}
I20260806 20:42:51.146421 12767 heartbeater.cc:347] Connected to a master server at 127.23.200.62:33191
I20260806 20:42:51.146739 12767 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:51.147619 12767 heartbeater.cc:511] Master 127.23.200.62:33191 requested a full tablet report, sending...
I20260806 20:42:51.149926 12579 ts_manager.cc:194] Registered new tserver with Master: ec45ea1ed3284ba487e65f8a260ff6e0 (127.23.200.1:43789)
I20260806 20:42:51.151968 12579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.1:47411
W20260806 20:42:51.469388 12771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:51.469636 12771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:51.469774 12771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:51.476832 12771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:51.476917 12771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:51.477097 12771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.2
I20260806 20:42:51.485672 12771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.200.2
--webserver_port=0
--tserver_master_addrs=127.23.200.62:33191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.2
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:51.486882 12771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:51.488593 12771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:51.501217 12776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:51.501262 12779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:51.501327 12777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:51.502458 12771 server_base.cc:1034] running on GCE node
I20260806 20:42:51.502985 12771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:51.503822 12771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:51.505026 12771 hybrid_clock.cc:648] HybridClock initialized: now 1786048971504977 us; error 38 us; skew 500 ppm
I20260806 20:42:51.505311 12771 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:51.507449 12771 webserver.cc:460] Webserver started at http://127.23.200.2:33847/ using document root <none> and password file <none>
I20260806 20:42:51.508119 12771 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:51.508204 12771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:51.508477 12771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:51.511188 12771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data/instance:
uuid: "05fa9a5b646f4c87a50ddf3339f73566"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "3bf5d8356071e54909953ef2c1dcd777"
server_key_iv: "d29d13d4d4d2d6fe7fe047867d43d026"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.511904 12771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance:
uuid: "05fa9a5b646f4c87a50ddf3339f73566"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "3bf5d8356071e54909953ef2c1dcd777"
server_key_iv: "d29d13d4d4d2d6fe7fe047867d43d026"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.516681 12771 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 20:42:51.520318 12785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.521627 12771 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:42:51.521791 12771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal
uuid: "05fa9a5b646f4c87a50ddf3339f73566"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "3bf5d8356071e54909953ef2c1dcd777"
server_key_iv: "d29d13d4d4d2d6fe7fe047867d43d026"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.522004 12771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:51.550612 12771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:51.551470 12771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:51.551784 12771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:51.552757 12771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:51.554198 12771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:51.554373 12771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.554509 12771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:51.554600 12771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.587330 12771 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.2:43377
I20260806 20:42:51.587426 12899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.2:43377 every 8 connection(s)
I20260806 20:42:51.588613 12771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/data/info.pb
I20260806 20:42:51.589620 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12771
I20260806 20:42:51.589841 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal/instance
I20260806 20:42:51.590583 24352 external_mini_cluster.cc:1428] Setting key 11dff21f4a5bcf6323bf14d8ebf6fd5d
I20260806 20:42:51.593895 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.200.3:0
--local_ip_for_outbound_sockets=127.23.200.3
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:33191
--builtin_ntp_servers=127.23.200.20:39239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 20:42:51.604451 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:51.591566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:58419 (local address 127.23.200.62:33191)
0806 20:42:51.591777 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:51.591784 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:51.593132 (+ 1348us) server_negotiation.cc:407] Connection header received
0806 20:42:51.596045 (+ 2913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:51.596050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:51.596126 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:51.596227 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:51.597569 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.598747 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.601207 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.601860 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.601990 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:51.603549 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:51.603553 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:51.603556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:51.603573 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:51.603603 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:51.603614 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:51.603683 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:51.603964 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 20:42:51.604095 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 20:42:51.605248 12902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:51.591879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:33191 (local address 127.23.200.2:58419)
0806 20:42:51.592978 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:51.593024 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:51.595732 (+ 2708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:51.596480 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:51.596494 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:51.596608 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:51.597379 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:51.597411 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.598924 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.598932 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:51.601052 (+ 2120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:51.601070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:51.602101 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:51.602110 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:51.602459 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:51.603191 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:51.603197 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:51.603260 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:51.603368 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:51.603709 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:51.603723 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:51.603733 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 20:42:51.603922 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 20:42:51.604229 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":325,"threads_started":1}
I20260806 20:42:51.606705 12900 heartbeater.cc:347] Connected to a master server at 127.23.200.62:33191
I20260806 20:42:51.606998 12900 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:51.607858 12900 heartbeater.cc:511] Master 127.23.200.62:33191 requested a full tablet report, sending...
I20260806 20:42:51.609304 12579 ts_manager.cc:194] Registered new tserver with Master: 05fa9a5b646f4c87a50ddf3339f73566 (127.23.200.2:43377)
I20260806 20:42:51.610096 12579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.2:58419
W20260806 20:42:51.923254 12903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:51.923506 12903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:51.923667 12903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:51.930891 12903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:51.930971 12903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:51.931154 12903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.200.3
I20260806 20:42:51.939841 12903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.200.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.200.3
--webserver_port=0
--tserver_master_addrs=127.23.200.62:33191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.200.3
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:51.941040 12903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:51.942819 12903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:51.955552 12909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:51.955518 12911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:51.955667 12908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:51.956331 12903 server_base.cc:1034] running on GCE node
I20260806 20:42:51.956874 12903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:51.957733 12903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:51.958953 12903 hybrid_clock.cc:648] HybridClock initialized: now 1786048971958920 us; error 52 us; skew 500 ppm
I20260806 20:42:51.959264 12903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 20:42:51.961364 12903 webserver.cc:460] Webserver started at http://127.23.200.3:35349/ using document root <none> and password file <none>
I20260806 20:42:51.962080 12903 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:51.962208 12903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:51.962569 12903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:51.965262 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data/instance:
uuid: "2e8ae5d3944e4570b107b27bce24021f"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "65abbd638ffe25e6a53ef0b59611c5ab"
server_key_iv: "2ed7971e0f5f9a404098cabc1dd145a6"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.966045 12903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance:
uuid: "2e8ae5d3944e4570b107b27bce24021f"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "65abbd638ffe25e6a53ef0b59611c5ab"
server_key_iv: "2ed7971e0f5f9a404098cabc1dd145a6"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.970840 12903 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 20:42:51.974115 12917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:51.975364 12903 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.004s sys 0.000s
I20260806 20:42:51.975538 12903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal
uuid: "2e8ae5d3944e4570b107b27bce24021f"
format_stamp: "Formatted at 2026-08-06 20:42:51 on dist-test-slave-tm1g"
server_key: "65abbd638ffe25e6a53ef0b59611c5ab"
server_key_iv: "2ed7971e0f5f9a404098cabc1dd145a6"
server_key_version: "encryptionkey@0"
I20260806 20:42:51.975736 12903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:52.006956 12903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:52.007761 12903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:52.008059 12903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:52.009044 12903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 20:42:52.010702 12903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 20:42:52.010814 12903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:52.010941 12903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 20:42:52.011042 12903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:52.056630 12903 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.3:38255
I20260806 20:42:52.056699 13033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.3:38255 every 8 connection(s)
I20260806 20:42:52.057943 12903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/data/info.pb
I20260806 20:42:52.063493 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 12903
I20260806 20:42:52.063670 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal/instance
I20260806 20:42:52.064289 24352 external_mini_cluster.cc:1428] Setting key 4f819749a5d40fcc8f14da9fbc3bef81
I20260806 20:42:52.071540 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.060631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.3:41563 (local address 127.23.200.62:33191)
0806 20:42:52.060891 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.060900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.062367 (+ 1467us) server_negotiation.cc:407] Connection header received
0806 20:42:52.064197 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.064202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.064295 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.064405 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:52.065882 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.066937 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.068753 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.069197 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.069349 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.070696 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:52.070702 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:52.070709 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:52.070731 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:52.070763 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:52.070775 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:52.070873 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.071084 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.071184 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 20:42:52.072454 13036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.061159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:33191 (local address 127.23.200.3:41563)
0806 20:42:52.062187 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:52.062231 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:52.063942 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:52.064709 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:52.064717 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:52.064831 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:42:52.065682 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.065709 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.067143 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.067148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.068595 (+ 1447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.068607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.069420 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.069427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.069642 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.070387 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 20:42:52.070390 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:42:52.070429 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 20:42:52.070497 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:42:52.070875 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:42:52.070888 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:42:52.070899 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 20:42:52.071073 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 20:42:52.071432 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":335,"threads_started":1}
I20260806 20:42:52.073724 13034 heartbeater.cc:347] Connected to a master server at 127.23.200.62:33191
I20260806 20:42:52.074019 13034 heartbeater.cc:464] Registering TS with master...
I20260806 20:42:52.074898 13034 heartbeater.cc:511] Master 127.23.200.62:33191 requested a full tablet report, sending...
I20260806 20:42:52.076298 12579 ts_manager.cc:194] Registered new tserver with Master: 2e8ae5d3944e4570b107b27bce24021f (127.23.200.3:38255)
I20260806 20:42:52.077129 12579 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.200.3:41563
I20260806 20:42:52.080350 24352 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 20:42:52.093442 12769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.086409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45802 (local address 127.23.200.62:33191)
0806 20:42:52.086634 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.086643 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.087016 (+ 373us) server_negotiation.cc:407] Connection header received
0806 20:42:52.087155 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.087163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.087255 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.087373 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:52.087870 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.089193 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.090599 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.091411 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.091557 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.092560 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:52.092569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:52.092576 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:52.092603 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:52.092647 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:52.092678 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:52.092786 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.092912 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.093026 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 20:42:52.097440 12579 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 20:42:52.104440 12579 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 20:42:52.147825 13046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.117812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.2:43377 (local address 127.0.0.1:42472)
0806 20:42:52.130821 (+ 13009us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:52.130864 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:52.131042 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:52.138666 (+ 7624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:52.138677 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:52.138725 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:52.139253 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.139269 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.141926 (+ 2657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.141935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.144206 (+ 2271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.144219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.146166 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.146176 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.146430 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.146508 (+ 78us) client_negotiation.cc:769] Sending connection context
0806 20:42:52.146613 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 20:42:52.146791 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12768,"thread_start_us":15464,"threads_started":1}
I20260806 20:42:52.149405 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.118003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.23.200.2:43377)
0806 20:42:52.137708 (+ 19705us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.137720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.137755 (+ 35us) server_negotiation.cc:407] Connection header received
0806 20:42:52.137869 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.137880 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.138181 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.139522 (+ 1341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:52.139890 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.141748 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.144408 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.145901 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.148579 (+ 2678us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.148812 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.148882 (+ 70us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.149034 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19494,"thread_start_us":20327,"threads_started":1}
I20260806 20:42:52.153345 12833 tablet_service.cc:1467] Processing CreateTablet for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab (DEFAULT_TABLE table=test-table [id=467acf3b90354f5f92a7cea7a6576e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:52.155010 12833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:52.155521 12767 heartbeater.cc:503] Master 127.23.200.62:33191 was elected leader, sending a full tablet report...
I20260806 20:42:52.157531 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.131032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46048 (local address 127.23.200.1:43789)
0806 20:42:52.143159 (+ 12127us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.143168 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.144771 (+ 1603us) server_negotiation.cc:407] Connection header received
0806 20:42:52.144923 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.144935 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.145288 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.145515 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:52.146683 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.148552 (+ 1869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.153624 (+ 5072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.155462 (+ 1838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.155599 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.155940 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.156864 (+ 924us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.157078 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11947,"thread_start_us":12125,"threads_started":1}
I20260806 20:42:52.157639 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.117983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:38255 (local address 127.0.0.1:59816)
0806 20:42:52.132172 (+ 14189us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:52.132189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:52.132296 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:52.146941 (+ 14645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:52.146945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:52.146970 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:52.147332 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.147347 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.149190 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.149197 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.152497 (+ 3300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.152517 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.154796 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.154803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.155007 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.155050 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 20:42:52.155121 (+ 71us) client_negotiation.cc:240] Negotiation successful
0806 20:42:52.155250 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14057,"mutex_wait_us":840,"thread_start_us":13921,"threads_started":1}
I20260806 20:42:52.158696 13050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.130117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:43789 (local address 127.0.0.1:46048)
0806 20:42:52.144640 (+ 14523us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:52.144660 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:52.144782 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:52.145882 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:52.145890 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:52.145923 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:52.146478 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.146494 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.148775 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.148783 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.153387 (+ 4604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.153405 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.156142 (+ 2737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.156151 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.156389 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.156439 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 20:42:52.156520 (+ 81us) client_negotiation.cc:240] Negotiation successful
0806 20:42:52.156697 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14376,"mutex_wait_us":117,"thread_start_us":14742,"threads_started":1}
I20260806 20:42:52.158793 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.118690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59816 (local address 127.23.200.3:38255)
0806 20:42:52.146394 (+ 27704us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.146411 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.146444 (+ 33us) server_negotiation.cc:407] Connection header received
0806 20:42:52.146537 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.146545 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.146760 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.146946 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:52.147537 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.149025 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.153132 (+ 4107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.154569 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.157888 (+ 3319us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.158150 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.158225 (+ 75us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.158414 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27484,"thread_start_us":28468,"threads_started":1}
I20260806 20:42:52.163333 12967 tablet_service.cc:1467] Processing CreateTablet for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab (DEFAULT_TABLE table=test-table [id=467acf3b90354f5f92a7cea7a6576e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:52.164096 12702 tablet_service.cc:1467] Processing CreateTablet for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab (DEFAULT_TABLE table=test-table [id=467acf3b90354f5f92a7cea7a6576e14]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 20:42:52.165107 12967 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:52.165469 12702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:52.183214 13052 tablet_bootstrap.cc:492] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Bootstrap starting.
I20260806 20:42:52.185822 13054 tablet_bootstrap.cc:492] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Bootstrap starting.
I20260806 20:42:52.188195 13052 tablet_bootstrap.cc:654] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:52.189509 13053 tablet_bootstrap.cc:492] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Bootstrap starting.
I20260806 20:42:52.189769 13054 tablet_bootstrap.cc:654] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:52.189847 13052 log.cc:824] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:52.191285 13054 log.cc:824] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:52.195812 13053 tablet_bootstrap.cc:654] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:52.197211 13053 log.cc:824] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:52.199664 13052 tablet_bootstrap.cc:492] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: No bootstrap required, opened a new log
I20260806 20:42:52.200081 13052 ts_tablet_manager.cc:1397] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.007s
I20260806 20:42:52.213466 13054 tablet_bootstrap.cc:492] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: No bootstrap required, opened a new log
I20260806 20:42:52.213943 13054 ts_tablet_manager.cc:1397] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.014s
I20260806 20:42:52.214615 13053 tablet_bootstrap.cc:492] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: No bootstrap required, opened a new log
I20260806 20:42:52.214938 13053 ts_tablet_manager.cc:1397] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.013s
I20260806 20:42:52.215515 13052 raft_consensus.cc:348] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.216101 13052 raft_consensus.cc:374] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:52.216269 13052 raft_consensus.cc:729] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05fa9a5b646f4c87a50ddf3339f73566, State: Initialized, Role: FOLLOWER
I20260806 20:42:52.217103 13052 consensus_queue.cc:260] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.225391 13052 ts_tablet_manager.cc:1428] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Time spent starting tablet: real 0.025s user 0.013s sys 0.008s
I20260806 20:42:52.225834 12900 heartbeater.cc:503] Master 127.23.200.62:33191 was elected leader, sending a full tablet report...
I20260806 20:42:52.228135 13054 raft_consensus.cc:348] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.228726 13054 raft_consensus.cc:374] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:52.228876 13054 raft_consensus.cc:729] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec45ea1ed3284ba487e65f8a260ff6e0, State: Initialized, Role: FOLLOWER
I20260806 20:42:52.228931 13053 raft_consensus.cc:348] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.229442 13053 raft_consensus.cc:374] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:52.229596 13053 raft_consensus.cc:729] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e8ae5d3944e4570b107b27bce24021f, State: Initialized, Role: FOLLOWER
I20260806 20:42:52.229717 13054 consensus_queue.cc:260] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.230408 13053 consensus_queue.cc:260] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.234354 13054 ts_tablet_manager.cc:1428] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Time spent starting tablet: real 0.020s user 0.016s sys 0.003s
I20260806 20:42:52.237588 13034 heartbeater.cc:503] Master 127.23.200.62:33191 was elected leader, sending a full tablet report...
I20260806 20:42:52.238605 13053 ts_tablet_manager.cc:1428] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Time spent starting tablet: real 0.024s user 0.012s sys 0.009s
W20260806 20:42:52.310071 13035 tablet.cc:2369] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:42:52.342072 12901 tablet.cc:2369] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 20:42:52.378263 12768 tablet.cc:2369] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:42:52.416136 13058 raft_consensus.cc:482] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 20:42:52.416559 13058 raft_consensus.cc:504] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.418288 13058 leader_election.cc:288] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e8ae5d3944e4570b107b27bce24021f (127.23.200.3:38255), ec45ea1ed3284ba487e65f8a260ff6e0 (127.23.200.1:43789)
I20260806 20:42:52.428431 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.419167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:39963 (local address 127.23.200.1:43789)
0806 20:42:52.419447 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.419457 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.420288 (+ 831us) server_negotiation.cc:407] Connection header received
0806 20:42:52.420504 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.420510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.420593 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.420697 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:52.421630 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.422808 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.426039 (+ 3231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.427379 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.427529 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.427602 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.428039 (+ 437us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.428178 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:42:52.428431 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.419185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.200.2:46307 (local address 127.23.200.3:38255)
0806 20:42:52.419496 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.419510 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.420290 (+ 780us) server_negotiation.cc:407] Connection header received
0806 20:42:52.420504 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.420510 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.420593 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.420697 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 20:42:52.421643 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.422826 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.426054 (+ 3228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.427356 (+ 1302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.427548 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.427608 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.428063 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.428188 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 20:42:52.428678 13062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.418666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.1:43789 (local address 127.23.200.2:39963)
0806 20:42:52.420122 (+ 1456us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:52.420143 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:52.420349 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:52.420728 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:52.420733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:52.420773 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:52.421403 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.421421 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.422985 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.422990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.425800 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.425817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.427601 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.427607 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.427808 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.427888 (+ 80us) client_negotiation.cc:769] Sending connection context
0806 20:42:52.427988 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 20:42:52.428182 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1285,"mutex_wait_us":283,"spinlock_wait_cycles":46976,"thread_start_us":1105,"threads_started":1}
I20260806 20:42:52.428701 13061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.418762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.3:38255 (local address 127.23.200.2:46307)
0806 20:42:52.420170 (+ 1408us) negotiation.cc:107] Waiting for socket to connect
0806 20:42:52.420187 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:42:52.420333 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:42:52.420728 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:42:52.420733 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:42:52.420786 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 20:42:52.421476 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.421490 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.422986 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.422990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.425801 (+ 2811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:42:52.425817 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.427592 (+ 1775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.427599 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:42:52.427808 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.427901 (+ 93us) client_negotiation.cc:769] Sending connection context
0806 20:42:52.427988 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 20:42:52.428182 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"mutex_wait_us":350,"spinlock_wait_cycles":36096,"thread_start_us":977,"threads_started":1}
I20260806 20:42:52.429747 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e814d8ad8d94bcab5fe2dae2d8bb3ab" candidate_uuid: "05fa9a5b646f4c87a50ddf3339f73566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" is_pre_election: true
I20260806 20:42:52.429955 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e814d8ad8d94bcab5fe2dae2d8bb3ab" candidate_uuid: "05fa9a5b646f4c87a50ddf3339f73566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8ae5d3944e4570b107b27bce24021f" is_pre_election: true
I20260806 20:42:52.430452 12721 raft_consensus.cc:2454] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fa9a5b646f4c87a50ddf3339f73566 in term 0.
I20260806 20:42:52.430495 12987 raft_consensus.cc:2454] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05fa9a5b646f4c87a50ddf3339f73566 in term 0.
I20260806 20:42:52.431260 12786 leader_election.cc:302] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 05fa9a5b646f4c87a50ddf3339f73566, 2e8ae5d3944e4570b107b27bce24021f, ec45ea1ed3284ba487e65f8a260ff6e0; no voters:
I20260806 20:42:52.431699 13058 raft_consensus.cc:2781] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 20:42:52.431860 13058 raft_consensus.cc:482] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 20:42:52.431978 13058 raft_consensus.cc:3037] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:52.434235 13058 raft_consensus.cc:504] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.434938 13058 leader_election.cc:288] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [CANDIDATE]: Term 1 election: Requested vote from peers 2e8ae5d3944e4570b107b27bce24021f (127.23.200.3:38255), ec45ea1ed3284ba487e65f8a260ff6e0 (127.23.200.1:43789)
I20260806 20:42:52.435449 12987 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e814d8ad8d94bcab5fe2dae2d8bb3ab" candidate_uuid: "05fa9a5b646f4c87a50ddf3339f73566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e8ae5d3944e4570b107b27bce24021f"
I20260806 20:42:52.435681 12987 raft_consensus.cc:3037] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:52.435653 12721 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e814d8ad8d94bcab5fe2dae2d8bb3ab" candidate_uuid: "05fa9a5b646f4c87a50ddf3339f73566" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0"
I20260806 20:42:52.435858 12721 raft_consensus.cc:3037] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:52.437947 12987 raft_consensus.cc:2454] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fa9a5b646f4c87a50ddf3339f73566 in term 1.
I20260806 20:42:52.438037 12721 raft_consensus.cc:2454] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05fa9a5b646f4c87a50ddf3339f73566 in term 1.
I20260806 20:42:52.438424 12787 leader_election.cc:302] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05fa9a5b646f4c87a50ddf3339f73566, 2e8ae5d3944e4570b107b27bce24021f; no voters:
I20260806 20:42:52.438760 13058 raft_consensus.cc:2781] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:52.439232 13058 raft_consensus.cc:686] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 1 LEADER]: Becoming Leader. State: Replica: 05fa9a5b646f4c87a50ddf3339f73566, State: Running, Role: LEADER
I20260806 20:42:52.439693 13058 consensus_queue.cc:237] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } }
I20260806 20:42:52.444166 12579 catalog_manager.cc:5495] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 reported cstate change: term changed from 0 to 1, leader changed from <none> to 05fa9a5b646f4c87a50ddf3339f73566 (127.23.200.2). New cstate: current_term: 1 leader_uuid: "05fa9a5b646f4c87a50ddf3339f73566" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05fa9a5b646f4c87a50ddf3339f73566" member_type: VOTER last_known_addr { host: "127.23.200.2" port: 43377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 } health_report { overall_health: UNKNOWN } } }
I20260806 20:42:52.505934 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.500364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.23.200.2:43377)
0806 20:42:52.500583 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.500589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.500690 (+ 101us) server_negotiation.cc:407] Connection header received
0806 20:42:52.500951 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.500955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.501056 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.501215 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:42:52.501612 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.502331 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.503721 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.504150 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.504295 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.504773 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:42:52.505321 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:42:52.505498 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.505580 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.505658 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 20:42:52.513947 12721 raft_consensus.cc:1261] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 1 FOLLOWER]: Refusing update from remote peer 05fa9a5b646f4c87a50ddf3339f73566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:52.514005 12987 raft_consensus.cc:1261] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 1 FOLLOWER]: Refusing update from remote peer 05fa9a5b646f4c87a50ddf3339f73566: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:42:52.514986 13063 consensus_queue.cc:1035] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec45ea1ed3284ba487e65f8a260ff6e0" member_type: VOTER last_known_addr { host: "127.23.200.1" port: 43789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:52.515319 13058 consensus_queue.cc:1035] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e8ae5d3944e4570b107b27bce24021f" member_type: VOTER last_known_addr { host: "127.23.200.3" port: 38255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:42:52.529534 13066 mvcc.cc:205] Tried to move back new op lower bound from 7315656591407611904 to 7315656591116668928. Current Snapshot: MvccSnapshot[applied={T|T < 7315656591407611904 or (T in {7315656591407611904})}]
I20260806 20:42:52.540169 12578 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:52.540457 12578 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 20:42:52.544054 12578 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df997f5658f94a6bb563b724e2af4dec: Sending DeleteTablet for 3 replicas of tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab
I20260806 20:42:52.545105 12833 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:52 UTC) from {username='slave'} at 127.0.0.1:42472
I20260806 20:42:52.545305 12967 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:52 UTC) from {username='slave'} at 127.0.0.1:59816
I20260806 20:42:52.545835 12702 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 20:42:52 UTC) from {username='slave'} at 127.0.0.1:46048
I20260806 20:42:52.546697 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12638
I20260806 20:42:52.546792 13077 tablet_replica.cc:330] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: stopping tablet replica
I20260806 20:42:52.547282 13077 raft_consensus.cc:2229] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:52.547832 13077 raft_consensus.cc:2258] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:52.555233 13077 ts_tablet_manager.cc:1905] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:52.556674 13076 tablet_replica.cc:330] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: stopping tablet replica
I20260806 20:42:52.557093 13076 raft_consensus.cc:2229] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 20:42:52.557587 13076 raft_consensus.cc:2258] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:52.559862 13075 tablet_replica.cc:330] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: stopping tablet replica
I20260806 20:42:52.560288 13075 raft_consensus.cc:2229] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 1 LEADER]: Raft consensus shutting down.
I20260806 20:42:52.561456 13075 raft_consensus.cc:2258] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 20:42:52.562932 13077 ts_tablet_manager.cc:1918] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:52.563095 13077 log.cc:1196] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-0/wal/wals/4e814d8ad8d94bcab5fe2dae2d8bb3ab
I20260806 20:42:52.563122 13076 ts_tablet_manager.cc:1905] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:52.563632 13077 ts_tablet_manager.cc:1939] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P ec45ea1ed3284ba487e65f8a260ff6e0: Deleting consensus metadata
I20260806 20:42:52.565510 12563 catalog_manager.cc:4841] TS ec45ea1ed3284ba487e65f8a260ff6e0 (127.23.200.1:43789): tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab (table test-table [id=467acf3b90354f5f92a7cea7a6576e14]) successfully deleted
I20260806 20:42:52.567293 13075 ts_tablet_manager.cc:1905] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 20:42:52.568323 13051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.548574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.23.200.1:43789)
0806 20:42:52.548783 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.548792 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.549124 (+ 332us) server_negotiation.cc:407] Connection header received
0806 20:42:52.549434 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.549442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.549566 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.549651 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:52.550481 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.551962 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.559217 (+ 7255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.559895 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.560000 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.566582 (+ 6582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:52.566593 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:52.566609 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:52.567331 (+ 722us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:52.567393 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:52.567418 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:52.567509 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.567834 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.567966 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 20:42:52.569638 12762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:52.570127 13076 ts_tablet_manager.cc:1918] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:52.570329 13076 log.cc:1196] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-2/wal/wals/4e814d8ad8d94bcab5fe2dae2d8bb3ab
I20260806 20:42:52.571357 13076 ts_tablet_manager.cc:1939] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 2e8ae5d3944e4570b107b27bce24021f: Deleting consensus metadata
I20260806 20:42:52.573849 12564 catalog_manager.cc:4841] TS 2e8ae5d3944e4570b107b27bce24021f (127.23.200.3:38255): tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab (table test-table [id=467acf3b90354f5f92a7cea7a6576e14]) successfully deleted
I20260806 20:42:52.579352 13075 ts_tablet_manager.cc:1918] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 20:42:52.579564 13075 log.cc:1196] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Deleting WAL directory at /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786048651485794-24352-0/minicluster-data/ts-1/wal/wals/4e814d8ad8d94bcab5fe2dae2d8bb3ab
I20260806 20:42:52.580154 13075 ts_tablet_manager.cc:1939] T 4e814d8ad8d94bcab5fe2dae2d8bb3ab P 05fa9a5b646f4c87a50ddf3339f73566: Deleting consensus metadata
I20260806 20:42:52.582188 12565 catalog_manager.cc:4841] TS 05fa9a5b646f4c87a50ddf3339f73566 (127.23.200.2:43377): tablet 4e814d8ad8d94bcab5fe2dae2d8bb3ab (table test-table [id=467acf3b90354f5f92a7cea7a6576e14]) successfully deleted
I20260806 20:42:52.678735 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12638
I20260806 20:42:52.697605 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12771
I20260806 20:42:52.704877 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.698468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42484 (local address 127.23.200.2:43377)
0806 20:42:52.698638 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.698643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.698693 (+ 50us) server_negotiation.cc:407] Connection header received
0806 20:42:52.698918 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.698924 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.699019 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.699139 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:52.699548 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.700368 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.701635 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.702086 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.702214 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.703448 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:52.703461 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:52.703476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:52.704108 (+ 632us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:52.704158 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:52.704177 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:52.704324 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.704444 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.704548 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:42:52.705631 12895 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:52.817749 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12771
I20260806 20:42:52.838819 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12903
I20260806 20:42:52.845443 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:52.839594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59820 (local address 127.23.200.3:38255)
0806 20:42:52.839763 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:52.839767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:52.839841 (+ 74us) server_negotiation.cc:407] Connection header received
0806 20:42:52.840008 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:52.840012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:52.840093 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:52.840178 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:52.840555 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.841282 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.842724 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:52.843103 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:52.843194 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:52.844143 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:52.844151 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:52.844165 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:52.844802 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:52.844850 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:52.844869 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:52.844953 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:52.845087 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 20:42:52.845177 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:42:52.846122 13029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:52.933182 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12903
I20260806 20:42:52.951311 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 12548
I20260806 20:42:52.952463 12609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:53.061363 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 12548
2026-08-06T20:42:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2980 ms)
[----------] 6 tests from EncryptionPolicyTest (21872 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:53 dist-test-slave-tm1g krb5kdc[13085](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:53 dist-test-slave-tm1g krb5kdc[13085](info): set up 2 sockets
Aug 06 20:42:53 dist-test-slave-tm1g krb5kdc[13085](info): commencing operation
krb5kdc: starting...
W20260806 20:42:54.202191 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:42:54 dist-test-slave-tm1g krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:42:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:42:54.475863 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34783
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:54.805577 13099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:54.805876 13099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:54.806059 13099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:54.813381 13099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:54.813493 13099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:54.813603 13099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:54.813707 13099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:54.813784 13099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:54.824167 13099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34783
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:54.825531 13099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:54.826910 13099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:54.838802 13104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:54.838842 13107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:54.838907 13105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:54.839792 13099 server_base.cc:1034] running on GCE node
I20260806 20:42:54.840564 13099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:54.841920 13099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:54.843146 13099 hybrid_clock.cc:648] HybridClock initialized: now 1786048974843113 us; error 45 us; skew 500 ppm
I20260806 20:42:54.843501 13099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:54 dist-test-slave-tm1g krb5kdc[13085](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048974, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:54.846561 13099 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:42:54.847941 13099 webserver.cc:460] Webserver started at http://127.0.0.1:32771/ using document root <none> and password file <none>
I20260806 20:42:54.848627 13099 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:54.848776 13099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:54.849092 13099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:54.852118 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "d362acd84d084da7935904b3a9fab8d5"
format_stamp: "Formatted at 2026-08-06 20:42:54 on dist-test-slave-tm1g"
server_key: "78c0f8e19a1c2bdf6cf48d3c35b9ea64"
server_key_iv: "efce8a3f03ab0ffb4902221d0d0f7422"
server_key_version: "encryptionkey@0"
I20260806 20:42:54.852896 13099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "d362acd84d084da7935904b3a9fab8d5"
format_stamp: "Formatted at 2026-08-06 20:42:54 on dist-test-slave-tm1g"
server_key: "78c0f8e19a1c2bdf6cf48d3c35b9ea64"
server_key_iv: "efce8a3f03ab0ffb4902221d0d0f7422"
server_key_version: "encryptionkey@0"
I20260806 20:42:54.857942 13099 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:42:54.861306 13114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:54.862779 13099 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260806 20:42:54.862965 13099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "d362acd84d084da7935904b3a9fab8d5"
format_stamp: "Formatted at 2026-08-06 20:42:54 on dist-test-slave-tm1g"
server_key: "78c0f8e19a1c2bdf6cf48d3c35b9ea64"
server_key_iv: "efce8a3f03ab0ffb4902221d0d0f7422"
server_key_version: "encryptionkey@0"
I20260806 20:42:54.863164 13099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:54.883339 13099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:54.887151 13099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:54.887428 13099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:54.905433 13099 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34783
I20260806 20:42:54.905455 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34783 every 8 connection(s)
I20260806 20:42:54.906831 13099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:54.909545 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13099
I20260806 20:42:54.909777 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:54.910624 24352 external_mini_cluster.cc:1428] Setting key 52ead2cbb03601f546dea7161f93c04e
I20260806 20:42:54.911015 13167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:54.925863 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5: Bootstrap starting.
Aug 06 20:42:54 dist-test-slave-tm1g krb5kdc[13085](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 20:42:54.928901 13167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:54.930263 13167 log.cc:824] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:54.933516 13167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5: No bootstrap required, opened a new log
I20260806 20:42:54.935729 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:54.912606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47089 (local address 127.0.0.1:34783)
0806 20:42:54.913382 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:54.913392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:54.913432 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:42:54.915910 (+ 2478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:54.915942 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:54.916315 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:54.916634 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:54.918472 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:54.919753 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:54.921574 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:54.922257 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:54.922474 (+ 217us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:54.927367 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:54.927374 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:54.927386 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:54.927434 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:54.931164 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:54.932018 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:54.932026 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:54.932033 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:54.932227 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:54.932645 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:54.932650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:54.932653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:54.933298 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:54.933622 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:54.934061 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 20:42:54.934422 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":381,"threads_started":1}
I20260806 20:42:54.939682 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:42:54.942420 13167 raft_consensus.cc:348] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d362acd84d084da7935904b3a9fab8d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34783 } }
I20260806 20:42:54.942839 13167 raft_consensus.cc:374] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:54.942971 13167 raft_consensus.cc:729] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d362acd84d084da7935904b3a9fab8d5, State: Initialized, Role: FOLLOWER
I20260806 20:42:54.943562 13167 consensus_queue.cc:260] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d362acd84d084da7935904b3a9fab8d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34783 } }
I20260806 20:42:54.943794 13167 raft_consensus.cc:388] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:54.943935 13167 raft_consensus.cc:482] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:54.944144 13167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:54.947544 13167 raft_consensus.cc:504] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d362acd84d084da7935904b3a9fab8d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34783 } }
I20260806 20:42:54.948110 13167 leader_election.cc:302] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d362acd84d084da7935904b3a9fab8d5; no voters:
I20260806 20:42:54.948741 13167 leader_election.cc:288] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:54.948858 13177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:54.950170 13177 raft_consensus.cc:686] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [term 1 LEADER]: Becoming Leader. State: Replica: d362acd84d084da7935904b3a9fab8d5, State: Running, Role: LEADER
I20260806 20:42:54.950755 13177 consensus_queue.cc:237] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d362acd84d084da7935904b3a9fab8d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34783 } }
I20260806 20:42:54.950973 13167 sys_catalog.cc:564] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:54.953579 13178 sys_catalog.cc:455] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d362acd84d084da7935904b3a9fab8d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d362acd84d084da7935904b3a9fab8d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34783 } } }
I20260806 20:42:54.953492 13179 sys_catalog.cc:455] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d362acd84d084da7935904b3a9fab8d5. Latest consensus state: current_term: 1 leader_uuid: "d362acd84d084da7935904b3a9fab8d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d362acd84d084da7935904b3a9fab8d5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34783 } } }
I20260806 20:42:54.953866 13179 sys_catalog.cc:458] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:54.953866 13178 sys_catalog.cc:458] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:54.963886 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:54.944176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54075 (local address 127.0.0.1:34783)
0806 20:42:54.944378 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:54.944386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:54.944942 (+ 556us) server_negotiation.cc:407] Connection header received
0806 20:42:54.945093 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:54.945097 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:54.945156 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:54.945228 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:54.947080 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:54.947859 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:54.949693 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:54.950505 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:54.950605 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:54.954252 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:54.954262 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:54.954270 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:42:54.954350 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:54.959119 (+ 4769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:54.960351 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:54.960360 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:54.960366 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:54.960568 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:42:54.961288 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:42:54.961296 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:42:54.961302 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:42:54.961840 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:54.962175 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:54.963119 (+ 944us) server_negotiation.cc:299] Negotiation successful
0806 20:42:54.963354 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:42:54.968708 13183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:54.973560 13183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:54.982962 13183 catalog_manager.cc:1347] Generated new cluster ID: 5b76be71ca8840028bfd6969dfbeb84b
I20260806 20:42:54.983084 13183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:54.998967 13183 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:55.000206 13183 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:55.007084 13183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d362acd84d084da7935904b3a9fab8d5: Generated new TSK 0
I20260806 20:42:55.007786 13183 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:42:55.015475 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13099
I20260806 20:42:55.016839 13161 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:42:55.106436 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13099
W20260806 20:42:55.125003 13083 connection.cc:535] client connection to 127.0.0.1:34783 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34783): Connection reset by peer (error 104)
2026-08-06T20:42:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2036 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:55 dist-test-slave-tm1g krb5kdc[13199](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:55 dist-test-slave-tm1g krb5kdc[13199](info): set up 2 sockets
Aug 06 20:42:55 dist-test-slave-tm1g krb5kdc[13199](info): commencing operation
krb5kdc: starting...
W20260806 20:42:56.227042 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:42:56 dist-test-slave-tm1g krb5kdc[13199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048976, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:42:56.465502 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:56.792065 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:56.792357 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:56.792498 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:56.799901 13212 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:56.800015 13212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:56.800130 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:56.800240 13212 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:56.800321 13212 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:56.811249 13212 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:56.812647 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:56.813997 13212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:56.826391 13217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:56.826292 13220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:56.826259 13218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:56.827486 13212 server_base.cc:1034] running on GCE node
I20260806 20:42:56.828262 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:56.829779 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:56.830982 13212 hybrid_clock.cc:648] HybridClock initialized: now 1786048976830939 us; error 35 us; skew 500 ppm
I20260806 20:42:56.831349 13212 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:56 dist-test-slave-tm1g krb5kdc[13199](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048976, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:56.834527 13212 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:42:56.835985 13212 webserver.cc:460] Webserver started at http://127.0.0.1:35769/ using document root <none> and password file <none>
I20260806 20:42:56.836706 13212 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:56.836870 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:56.837180 13212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:56.840401 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "02ee6a04ee1d4a2484af119788d08215"
format_stamp: "Formatted at 2026-08-06 20:42:56 on dist-test-slave-tm1g"
server_key: "59a03b1e8a8bd0c6a93e09f603811bd8"
server_key_iv: "a445a290c640dd598cdaac943a8fc63d"
server_key_version: "encryptionkey@0"
I20260806 20:42:56.841205 13212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "02ee6a04ee1d4a2484af119788d08215"
format_stamp: "Formatted at 2026-08-06 20:42:56 on dist-test-slave-tm1g"
server_key: "59a03b1e8a8bd0c6a93e09f603811bd8"
server_key_iv: "a445a290c640dd598cdaac943a8fc63d"
server_key_version: "encryptionkey@0"
I20260806 20:42:56.846514 13212 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 20:42:56.849881 13227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:56.851174 13212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:42:56.851366 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "02ee6a04ee1d4a2484af119788d08215"
format_stamp: "Formatted at 2026-08-06 20:42:56 on dist-test-slave-tm1g"
server_key: "59a03b1e8a8bd0c6a93e09f603811bd8"
server_key_iv: "a445a290c640dd598cdaac943a8fc63d"
server_key_version: "encryptionkey@0"
I20260806 20:42:56.851572 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:56.864046 13212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:56.868095 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:56.868376 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:56.887255 13212 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44273
I20260806 20:42:56.887272 13279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44273 every 8 connection(s)
I20260806 20:42:56.888710 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:56.892318 13280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:56.897635 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13212
I20260806 20:42:56.897836 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:56.898274 24352 external_mini_cluster.cc:1428] Setting key 738a1134a0a1faec831423dc29ab31f2
I20260806 20:42:56.903138 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215: Bootstrap starting.
I20260806 20:42:56.906713 13280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:56.907938 13280 log.cc:824] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:56.911356 13280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215: No bootstrap required, opened a new log
I20260806 20:42:56.912746 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:56.900393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55621 (local address 127.0.0.1:44273)
0806 20:42:56.901246 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:56.901260 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:56.901300 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:42:56.902995 (+ 1695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:56.903017 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:56.903491 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:56.903799 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:56.904448 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:56.905788 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:56.907741 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:56.908586 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:56.908798 (+ 212us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:56.909980 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:56.909991 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:56.910007 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:56.910737 (+ 730us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:56.910829 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:56.910863 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:56.911027 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:56.911262 (+ 235us) server_negotiation.cc:299] Negotiation successful
0806 20:42:56.911438 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":421,"threads_started":1}
I20260806 20:42:56.916627 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:42:56.920084 13280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ee6a04ee1d4a2484af119788d08215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44273 } }
I20260806 20:42:56.920440 13280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:56.920557 13280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02ee6a04ee1d4a2484af119788d08215, State: Initialized, Role: FOLLOWER
I20260806 20:42:56.921205 13280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ee6a04ee1d4a2484af119788d08215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44273 } }
I20260806 20:42:56.921509 13280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:56.921657 13280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:56.921860 13280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:56.924801 13280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ee6a04ee1d4a2484af119788d08215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44273 } }
I20260806 20:42:56.925374 13280 leader_election.cc:302] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02ee6a04ee1d4a2484af119788d08215; no voters:
I20260806 20:42:56.926050 13280 leader_election.cc:288] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:56.926155 13290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:56.927296 13290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [term 1 LEADER]: Becoming Leader. State: Replica: 02ee6a04ee1d4a2484af119788d08215, State: Running, Role: LEADER
I20260806 20:42:56.927939 13290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ee6a04ee1d4a2484af119788d08215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44273 } }
I20260806 20:42:56.928555 13280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:56.928826 13283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:56.921095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36227 (local address 127.0.0.1:44273)
0806 20:42:56.921278 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:56.921285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:56.921908 (+ 623us) server_negotiation.cc:407] Connection header received
0806 20:42:56.922095 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:56.922100 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:56.922184 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:56.922294 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:56.922856 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:56.924015 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:56.925376 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:42:56.926156 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:42:56.926280 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:42:56.927734 (+ 1454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:56.927741 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:56.927746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:56.927767 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:56.927821 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:56.927842 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:56.927977 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:56.928385 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 20:42:56.928435 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 20:42:56.931071 13292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02ee6a04ee1d4a2484af119788d08215. Latest consensus state: current_term: 1 leader_uuid: "02ee6a04ee1d4a2484af119788d08215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ee6a04ee1d4a2484af119788d08215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44273 } } }
I20260806 20:42:56.931448 13292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:56.932272 13291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02ee6a04ee1d4a2484af119788d08215" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02ee6a04ee1d4a2484af119788d08215" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44273 } } }
I20260806 20:42:56.932507 13298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:56.932655 13291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215 [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:56.938943 13298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:56.947439 13298 catalog_manager.cc:1347] Generated new cluster ID: 589b8ac6abba42c582a4a646e5f6f49c
I20260806 20:42:56.947574 13298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:56.956876 13298 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:56.958364 13298 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:56.963539 13298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02ee6a04ee1d4a2484af119788d08215: Generated new TSK 0
I20260806 20:42:56.964385 13298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:42:56.979610 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13212
W20260806 20:42:56.980751 13275 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55621
W20260806 20:42:56.981551 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13212: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:42:56.981735 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13212
W20260806 20:42:56.999631 13197 connection.cc:535] client connection to 127.0.0.1:44273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44273): Connection reset by peer (error 104)
2026-08-06T20:42:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1874 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:57 dist-test-slave-tm1g krb5kdc[13312](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:57 dist-test-slave-tm1g krb5kdc[13312](info): set up 2 sockets
Aug 06 20:42:57 dist-test-slave-tm1g krb5kdc[13312](info): commencing operation
krb5kdc: starting...
W20260806 20:42:58.106594 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:42:58 dist-test-slave-tm1g krb5kdc[13312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:42:58.368247 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38815
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:42:58.697479 13325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:42:58.697758 13325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:42:58.697856 13325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:42:58.705036 13325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:42:58.705137 13325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:42:58.705183 13325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:42:58.705245 13325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:42:58.705286 13325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:42:58.714859 13325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38815
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:42:58.716033 13325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:42:58.717260 13325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:42:58.729135 13330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:58.729175 13333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:42:58.729225 13331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:42:58.735416 13325 server_base.cc:1034] running on GCE node
I20260806 20:42:58.736127 13325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:42:58.737427 13325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:42:58.738654 13325 hybrid_clock.cc:648] HybridClock initialized: now 1786048978738618 us; error 49 us; skew 500 ppm
I20260806 20:42:58.738991 13325 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:42:58 dist-test-slave-tm1g krb5kdc[13312](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048978, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:42:58.745750 13325 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:42:58.747296 13325 webserver.cc:460] Webserver started at http://127.0.0.1:46715/ using document root <none> and password file <none>
I20260806 20:42:58.747998 13325 fs_manager.cc:359] Metadata directory not provided
I20260806 20:42:58.748137 13325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:42:58.748466 13325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:42:58.751400 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "892c14fdb2034f6b865bdd468369a63b"
format_stamp: "Formatted at 2026-08-06 20:42:58 on dist-test-slave-tm1g"
server_key: "1e5119e45524224bed2ca7452aace9d5"
server_key_iv: "a8252223a7448a11e0b525418b293cbe"
server_key_version: "encryptionkey@0"
I20260806 20:42:58.752146 13325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "892c14fdb2034f6b865bdd468369a63b"
format_stamp: "Formatted at 2026-08-06 20:42:58 on dist-test-slave-tm1g"
server_key: "1e5119e45524224bed2ca7452aace9d5"
server_key_iv: "a8252223a7448a11e0b525418b293cbe"
server_key_version: "encryptionkey@0"
I20260806 20:42:58.757126 13325 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:42:58.760475 13340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:42:58.761713 13325 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 20:42:58.761889 13325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "892c14fdb2034f6b865bdd468369a63b"
format_stamp: "Formatted at 2026-08-06 20:42:58 on dist-test-slave-tm1g"
server_key: "1e5119e45524224bed2ca7452aace9d5"
server_key_iv: "a8252223a7448a11e0b525418b293cbe"
server_key_version: "encryptionkey@0"
I20260806 20:42:58.762084 13325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:42:58.767760 13325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:42:58.772027 13325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:42:58.772321 13325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:42:58.790920 13325 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38815
I20260806 20:42:58.790969 13392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38815 every 8 connection(s)
I20260806 20:42:58.792351 13325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:42:58.792896 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13325
I20260806 20:42:58.793097 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:42:58.793712 24352 external_mini_cluster.cc:1428] Setting key 347b33ce7f0e0861c7068d6f0086c3ff
I20260806 20:42:58.796998 13393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:42:58.802538 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:58.795564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45591 (local address 127.0.0.1:38815)
0806 20:42:58.796369 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:58.796384 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:58.796443 (+ 59us) server_negotiation.cc:407] Connection header received
0806 20:42:58.798794 (+ 2351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:58.798820 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:58.799223 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:58.799644 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:58.800620 (+ 976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:58.800629 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:58.800650 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:58.800666 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:58.800755 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:58.800786 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:58.801000 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:58.801123 (+ 123us) server_negotiation.cc:299] Negotiation successful
0806 20:42:58.801322 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":324,"threads_started":1}
I20260806 20:42:58.806144 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:42:58.810344 13393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b: Bootstrap starting.
I20260806 20:42:58.813548 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:42:58.811024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37991 (local address 127.0.0.1:38815)
0806 20:42:58.811218 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 20:42:58.811226 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:42:58.811682 (+ 456us) server_negotiation.cc:407] Connection header received
0806 20:42:58.811834 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:42:58.811838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:42:58.811885 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:42:58.811974 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:42:58.812903 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:42:58.812907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:42:58.812910 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:42:58.812924 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:42:58.812967 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:42:58.812981 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:42:58.813065 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 20:42:58.813204 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 20:42:58.813247 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 20:42:58.814425 13393 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b: Neither blocks nor log segments found. Creating new log.
I20260806 20:42:58.815523 13393 log.cc:824] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b: Log is configured to *not* fsync() on all Append() calls
I20260806 20:42:58.818015 13393 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b: No bootstrap required, opened a new log
I20260806 20:42:58.825289 13393 raft_consensus.cc:348] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892c14fdb2034f6b865bdd468369a63b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260806 20:42:58.825639 13393 raft_consensus.cc:374] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:42:58.825709 13393 raft_consensus.cc:729] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 892c14fdb2034f6b865bdd468369a63b, State: Initialized, Role: FOLLOWER
I20260806 20:42:58.826247 13393 consensus_queue.cc:260] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892c14fdb2034f6b865bdd468369a63b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260806 20:42:58.826503 13393 raft_consensus.cc:388] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:42:58.826603 13393 raft_consensus.cc:482] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:42:58.826728 13393 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:42:58.828732 13393 raft_consensus.cc:504] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892c14fdb2034f6b865bdd468369a63b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260806 20:42:58.829166 13393 leader_election.cc:302] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 892c14fdb2034f6b865bdd468369a63b; no voters:
I20260806 20:42:58.829761 13393 leader_election.cc:288] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:42:58.829913 13404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:42:58.830750 13404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [term 1 LEADER]: Becoming Leader. State: Replica: 892c14fdb2034f6b865bdd468369a63b, State: Running, Role: LEADER
I20260806 20:42:58.831305 13404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892c14fdb2034f6b865bdd468369a63b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } }
I20260806 20:42:58.831796 13393 sys_catalog.cc:564] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:42:58.834172 13406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 892c14fdb2034f6b865bdd468369a63b. Latest consensus state: current_term: 1 leader_uuid: "892c14fdb2034f6b865bdd468369a63b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892c14fdb2034f6b865bdd468369a63b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } } }
I20260806 20:42:58.834720 13406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:58.835399 13405 sys_catalog.cc:455] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "892c14fdb2034f6b865bdd468369a63b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "892c14fdb2034f6b865bdd468369a63b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38815 } } }
I20260806 20:42:58.835811 13405 sys_catalog.cc:458] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b [sys.catalog]: This master's current role is: LEADER
I20260806 20:42:58.837373 13412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:42:58.841914 13412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:42:58.850273 13412 catalog_manager.cc:1347] Generated new cluster ID: 2662d7deca424c929a9570716194b904
I20260806 20:42:58.850417 13412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:42:58.888945 13412 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:42:58.889393 13412 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:42:58.896430 13412 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 892c14fdb2034f6b865bdd468369a63b: Generated new TSK 0
I20260806 20:42:58.897192 13412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:42:58.904878 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13325
W20260806 20:42:58.906334 13388 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45591
W20260806 20:42:58.907027 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13325: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:42:58.907191 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13325
2026-08-06T20:42:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:42:59 dist-test-slave-tm1g krb5kdc[13425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:42:59 dist-test-slave-tm1g krb5kdc[13425](info): set up 2 sockets
Aug 06 20:42:59 dist-test-slave-tm1g krb5kdc[13425](info): commencing operation
krb5kdc: starting...
W20260806 20:43:00.021792 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:00 dist-test-slave-tm1g krb5kdc[13425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:00.266891 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38161
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:00.593568 13438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:00.593840 13438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:00.593940 13438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:00.601090 13438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:00.601197 13438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:00.601241 13438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:00.601306 13438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:00.601348 13438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:00.610950 13438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:00.612123 13438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:00.613340 13438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:00.624722 13444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:00.624812 13443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:00.624953 13446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:00.625849 13438 server_base.cc:1034] running on GCE node
I20260806 20:43:00.626636 13438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:00.627930 13438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:00.629129 13438 hybrid_clock.cc:648] HybridClock initialized: now 1786048980629092 us; error 35 us; skew 500 ppm
I20260806 20:43:00.629467 13438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:00 dist-test-slave-tm1g krb5kdc[13425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:00.632560 13438 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:00.633956 13438 webserver.cc:460] Webserver started at http://127.0.0.1:40025/ using document root <none> and password file <none>
I20260806 20:43:00.634683 13438 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:00.634831 13438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:00.635123 13438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:00.637944 13438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "cff3ae1d885e4e8cb945d84446be807c"
format_stamp: "Formatted at 2026-08-06 20:43:00 on dist-test-slave-tm1g"
server_key: "00c5c7e2119b9a3987828bef84a1f370"
server_key_iv: "e9ebb3c731f2b8d23afbffa8f7302c87"
server_key_version: "encryptionkey@0"
I20260806 20:43:00.638726 13438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "cff3ae1d885e4e8cb945d84446be807c"
format_stamp: "Formatted at 2026-08-06 20:43:00 on dist-test-slave-tm1g"
server_key: "00c5c7e2119b9a3987828bef84a1f370"
server_key_iv: "e9ebb3c731f2b8d23afbffa8f7302c87"
server_key_version: "encryptionkey@0"
I20260806 20:43:00.643778 13438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 20:43:00.647064 13454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:00.648499 13438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:43:00.648635 13438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "cff3ae1d885e4e8cb945d84446be807c"
format_stamp: "Formatted at 2026-08-06 20:43:00 on dist-test-slave-tm1g"
server_key: "00c5c7e2119b9a3987828bef84a1f370"
server_key_iv: "e9ebb3c731f2b8d23afbffa8f7302c87"
server_key_version: "encryptionkey@0"
I20260806 20:43:00.648797 13438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:00.663625 13438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:00.667400 13438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:00.667629 13438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:00.685731 13438 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38161
I20260806 20:43:00.685774 13505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38161 every 8 connection(s)
I20260806 20:43:00.687088 13438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:00.688383 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13438
I20260806 20:43:00.688578 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:00.689024 24352 external_mini_cluster.cc:1428] Setting key 2aefedc83bb1b013ada8a1c5ae8bd95a
I20260806 20:43:00.691470 13506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 20:43:00 dist-test-slave-tm1g krb5kdc[13425](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 20:43:00.704028 13506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c: Bootstrap starting.
I20260806 20:43:00.708011 13506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:00.709363 13506 log.cc:824] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:00.712220 13506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c: No bootstrap required, opened a new log
I20260806 20:43:00.713341 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:00.690773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56275 (local address 127.0.0.1:38161)
0806 20:43:00.691571 (+ 798us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:00.691585 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:00.691651 (+ 66us) server_negotiation.cc:407] Connection header received
0806 20:43:00.693932 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:00.693958 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:00.694488 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:00.694929 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:00.696199 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:00.698276 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:00.699591 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:00.700646 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:00.700844 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:00.705018 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:00.705029 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:00.705047 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:00.705097 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:00.709139 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:00.709891 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:00.709903 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:00.709914 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:00.710212 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:00.710676 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:00.710684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:00.710687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:00.711180 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:00.711496 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:00.711968 (+ 472us) server_negotiation.cc:299] Negotiation successful
0806 20:43:00.712437 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":321,"threads_started":1}
I20260806 20:43:00.716943 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:00.720239 13506 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff3ae1d885e4e8cb945d84446be807c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260806 20:43:00.720603 13506 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:00.720708 13506 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff3ae1d885e4e8cb945d84446be807c, State: Initialized, Role: FOLLOWER
I20260806 20:43:00.721318 13506 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff3ae1d885e4e8cb945d84446be807c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260806 20:43:00.721597 13506 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:00.721747 13506 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:00.721936 13506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:00.724043 13506 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff3ae1d885e4e8cb945d84446be807c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260806 20:43:00.724521 13506 leader_election.cc:302] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff3ae1d885e4e8cb945d84446be807c; no voters:
I20260806 20:43:00.725185 13506 leader_election.cc:288] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:00.725309 13516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:00.727841 13506 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:00.734630 13516 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [term 1 LEADER]: Becoming Leader. State: Replica: cff3ae1d885e4e8cb945d84446be807c, State: Running, Role: LEADER
I20260806 20:43:00.735188 13516 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff3ae1d885e4e8cb945d84446be807c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } }
I20260806 20:43:00.736238 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:00.721224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33447 (local address 127.0.0.1:38161)
0806 20:43:00.721415 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:00.721421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:00.721912 (+ 491us) server_negotiation.cc:407] Connection header received
0806 20:43:00.722090 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:00.722096 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:00.722164 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:00.722262 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:00.723641 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:00.724411 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:00.725931 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:00.726595 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:00.726669 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:00.729200 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:00.729205 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:00.729210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:00.729250 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:00.733392 (+ 4142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:00.734100 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:00.734106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:00.734109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:00.734293 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:00.734729 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:00.734734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:00.734737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:00.735080 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:00.735388 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:00.735684 (+ 296us) server_negotiation.cc:299] Negotiation successful
0806 20:43:00.735907 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 20:43:00.738222 13517 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff3ae1d885e4e8cb945d84446be807c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff3ae1d885e4e8cb945d84446be807c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } } }
I20260806 20:43:00.738379 13518 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff3ae1d885e4e8cb945d84446be807c. Latest consensus state: current_term: 1 leader_uuid: "cff3ae1d885e4e8cb945d84446be807c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff3ae1d885e4e8cb945d84446be807c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38161 } } }
I20260806 20:43:00.738794 13517 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:00.738826 13518 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c [sys.catalog]: This master's current role is: LEADER
W20260806 20:43:00.744551 13531 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:43:00.744683 13531 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:43:00.744830 13532 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:00.746270 13532 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:00.754541 13532 catalog_manager.cc:1347] Generated new cluster ID: a4b01d724abc413b963846df4e06f597
I20260806 20:43:00.754659 13532 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:00.764875 13532 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:00.766659 13532 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:00.789606 13532 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff3ae1d885e4e8cb945d84446be807c: Generated new TSK 0
I20260806 20:43:00.790802 13532 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:00.808262 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13438
I20260806 20:43:00.809592 13501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:00.901167 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13438
2026-08-06T20:43:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1990 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:00 dist-test-slave-tm1g krb5kdc[13539](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:01 dist-test-slave-tm1g krb5kdc[13539](info): set up 2 sockets
Aug 06 20:43:01 dist-test-slave-tm1g krb5kdc[13539](info): commencing operation
krb5kdc: starting...
W20260806 20:43:02.030676 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:02 dist-test-slave-tm1g krb5kdc[13539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:02.299158 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41229
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:02.626642 13552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:02.626950 13552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:02.627135 13552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:02.634431 13552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:02.634544 13552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:02.634656 13552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:02.634732 13552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:02.634776 13552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:02.644842 13552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:02.646189 13552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:02.647573 13552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:02.659726 13558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:02.659940 13557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:02.659760 13560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:02.661160 13552 server_base.cc:1034] running on GCE node
I20260806 20:43:02.662046 13552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:02.663573 13552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:02.664777 13552 hybrid_clock.cc:648] HybridClock initialized: now 1786048982664739 us; error 35 us; skew 500 ppm
I20260806 20:43:02.665138 13552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:02 dist-test-slave-tm1g krb5kdc[13539](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:02.668334 13552 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:02.669894 13552 webserver.cc:460] Webserver started at http://127.0.0.1:34637/ using document root <none> and password file <none>
I20260806 20:43:02.670691 13552 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:02.670823 13552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:02.671149 13552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:02.674199 13552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "a50a6f98476a4690abc0dabe86535e41"
format_stamp: "Formatted at 2026-08-06 20:43:02 on dist-test-slave-tm1g"
server_key: "b2f85e370f29afc2f47cc0e96762f0fb"
server_key_iv: "06270014cdad691323800aa238fd5f49"
server_key_version: "encryptionkey@0"
I20260806 20:43:02.675083 13552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "a50a6f98476a4690abc0dabe86535e41"
format_stamp: "Formatted at 2026-08-06 20:43:02 on dist-test-slave-tm1g"
server_key: "b2f85e370f29afc2f47cc0e96762f0fb"
server_key_iv: "06270014cdad691323800aa238fd5f49"
server_key_version: "encryptionkey@0"
I20260806 20:43:02.680382 13552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:43:02.683938 13567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:02.685366 13552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:43:02.685565 13552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "a50a6f98476a4690abc0dabe86535e41"
format_stamp: "Formatted at 2026-08-06 20:43:02 on dist-test-slave-tm1g"
server_key: "b2f85e370f29afc2f47cc0e96762f0fb"
server_key_iv: "06270014cdad691323800aa238fd5f49"
server_key_version: "encryptionkey@0"
I20260806 20:43:02.685792 13552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:02.735626 13552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:02.739816 13552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:02.740147 13552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:02.759275 13552 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41229
I20260806 20:43:02.759315 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41229 every 8 connection(s)
I20260806 20:43:02.760666 13552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:02.762254 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13552
I20260806 20:43:02.762467 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:02.763052 24352 external_mini_cluster.cc:1428] Setting key 98d2741d250385e8de56eac34d48dad1
I20260806 20:43:02.765345 13620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:02.776551 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:02.764866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46801 (local address 127.0.0.1:41229)
0806 20:43:02.765716 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:02.765728 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:02.765768 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:43:02.767502 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:02.767523 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:02.767905 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:02.768281 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:02.768901 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:02.770233 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:02.772173 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:02.772819 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:02.772944 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:02.774232 (+ 1288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:02.774239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:02.774254 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:02.774871 (+ 617us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:02.774944 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:02.774971 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:02.775068 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:02.775291 (+ 223us) server_negotiation.cc:299] Negotiation successful
0806 20:43:02.775588 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":398,"threads_started":1}
I20260806 20:43:02.778033 13620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41: Bootstrap starting.
I20260806 20:43:02.780342 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:02.781399 13620 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:02.782423 13620 log.cc:824] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:02.785238 13620 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41: No bootstrap required, opened a new log
I20260806 20:43:02.791101 13623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:02.784704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52637 (local address 127.0.0.1:41229)
0806 20:43:02.784884 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:02.784890 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:02.785371 (+ 481us) server_negotiation.cc:407] Connection header received
0806 20:43:02.785589 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:02.785593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:02.785653 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:02.785766 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:02.786278 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:02.787069 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:02.788387 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:02.788918 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:02.789063 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:02.790284 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:02.790289 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:02.790295 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:02.790355 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:02.790407 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:02.790427 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:02.790531 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:02.790700 (+ 169us) server_negotiation.cc:299] Negotiation successful
0806 20:43:02.790788 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 20:43:02.793954 13620 raft_consensus.cc:348] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a6f98476a4690abc0dabe86535e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41229 } }
I20260806 20:43:02.794342 13620 raft_consensus.cc:374] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:02.794473 13620 raft_consensus.cc:729] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50a6f98476a4690abc0dabe86535e41, State: Initialized, Role: FOLLOWER
I20260806 20:43:02.795109 13620 consensus_queue.cc:260] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a6f98476a4690abc0dabe86535e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41229 } }
I20260806 20:43:02.795343 13620 raft_consensus.cc:388] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:02.795513 13620 raft_consensus.cc:482] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:02.795697 13620 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:02.798174 13620 raft_consensus.cc:504] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a6f98476a4690abc0dabe86535e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41229 } }
I20260806 20:43:02.798684 13620 leader_election.cc:302] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a50a6f98476a4690abc0dabe86535e41; no voters:
I20260806 20:43:02.799245 13620 leader_election.cc:288] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:02.799417 13630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:02.800097 13630 raft_consensus.cc:686] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [term 1 LEADER]: Becoming Leader. State: Replica: a50a6f98476a4690abc0dabe86535e41, State: Running, Role: LEADER
I20260806 20:43:02.800724 13630 consensus_queue.cc:237] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a6f98476a4690abc0dabe86535e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41229 } }
I20260806 20:43:02.801412 13620 sys_catalog.cc:564] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:02.803627 13631 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a50a6f98476a4690abc0dabe86535e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a6f98476a4690abc0dabe86535e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41229 } } }
I20260806 20:43:02.804016 13631 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:02.803731 13632 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a50a6f98476a4690abc0dabe86535e41. Latest consensus state: current_term: 1 leader_uuid: "a50a6f98476a4690abc0dabe86535e41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50a6f98476a4690abc0dabe86535e41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41229 } } }
I20260806 20:43:02.804180 13632 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:02.811254 13636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:02.816937 13636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:02.825814 13636 catalog_manager.cc:1347] Generated new cluster ID: 9a382139b5494719bc8c1847c85ad2b4
I20260806 20:43:02.825930 13636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:02.844310 13636 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:02.846187 13636 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:02.861302 13636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a50a6f98476a4690abc0dabe86535e41: Generated new TSK 0
I20260806 20:43:02.862506 13636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:02.870025 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13552
W20260806 20:43:02.871699 13615 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46801
W20260806 20:43:02.872464 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13552: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:02.872613 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13552
2026-08-06T20:43:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1974 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:02 dist-test-slave-tm1g krb5kdc[13651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:02 dist-test-slave-tm1g krb5kdc[13651](info): set up 2 sockets
Aug 06 20:43:02 dist-test-slave-tm1g krb5kdc[13651](info): commencing operation
krb5kdc: starting...
W20260806 20:43:04.006279 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:04 dist-test-slave-tm1g krb5kdc[13651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048984, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:04.275611 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40265
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:04.604838 13665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:04.605106 13665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:04.605273 13665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:04.612223 13665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:04.612317 13665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:04.612419 13665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:04.612502 13665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:04.612564 13665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:04.621941 13665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:04.623180 13665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:04.624490 13665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:04.640563 13673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:04.640540 13671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:04.641163 13665 server_base.cc:1034] running on GCE node
W20260806 20:43:04.641314 13670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:04.642160 13665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:04.643520 13665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:04.644851 13665 hybrid_clock.cc:648] HybridClock initialized: now 1786048984644805 us; error 43 us; skew 500 ppm
I20260806 20:43:04.645217 13665 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:04 dist-test-slave-tm1g krb5kdc[13651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048984, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:04.655661 13665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:04.657335 13665 webserver.cc:460] Webserver started at http://127.0.0.1:45045/ using document root <none> and password file <none>
I20260806 20:43:04.658036 13665 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:04.658185 13665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:04.658599 13665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:04.661563 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "5cf92c6f55b4403595e273b7e9f05eff"
format_stamp: "Formatted at 2026-08-06 20:43:04 on dist-test-slave-tm1g"
server_key: "b18b1faffd5e9f33906faed3f9c77f5e"
server_key_iv: "9406092f32f78a29590519e06bddcc51"
server_key_version: "encryptionkey@0"
I20260806 20:43:04.662401 13665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "5cf92c6f55b4403595e273b7e9f05eff"
format_stamp: "Formatted at 2026-08-06 20:43:04 on dist-test-slave-tm1g"
server_key: "b18b1faffd5e9f33906faed3f9c77f5e"
server_key_iv: "9406092f32f78a29590519e06bddcc51"
server_key_version: "encryptionkey@0"
I20260806 20:43:04.667760 13665 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260806 20:43:04.671259 13680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:04.672616 13665 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:43:04.672801 13665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "5cf92c6f55b4403595e273b7e9f05eff"
format_stamp: "Formatted at 2026-08-06 20:43:04 on dist-test-slave-tm1g"
server_key: "b18b1faffd5e9f33906faed3f9c77f5e"
server_key_iv: "9406092f32f78a29590519e06bddcc51"
server_key_version: "encryptionkey@0"
I20260806 20:43:04.673007 13665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:04.678712 13665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:04.682811 13665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:04.683109 13665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:04.703481 13665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40265
I20260806 20:43:04.703585 13731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40265 every 8 connection(s)
I20260806 20:43:04.704898 13665 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:04.708359 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13665
I20260806 20:43:04.708573 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:04.709024 24352 external_mini_cluster.cc:1428] Setting key 9ba13585d774b519ba4584f9d3ed5574
I20260806 20:43:04.709888 13733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:04.717828 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:04.710808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50147 (local address 127.0.0.1:40265)
0806 20:43:04.711697 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:04.711715 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:04.711781 (+ 66us) server_negotiation.cc:407] Connection header received
0806 20:43:04.713867 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:04.713894 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:04.714368 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:04.714796 (+ 428us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:04.715904 (+ 1108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:04.715912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:04.715935 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:04.715957 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:04.716054 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:04.716082 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:04.716296 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:04.716429 (+ 133us) server_negotiation.cc:299] Negotiation successful
0806 20:43:04.716615 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":378,"threads_started":1}
I20260806 20:43:04.722105 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:04.724778 13733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff: Bootstrap starting.
I20260806 20:43:04.728461 13733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:04.729802 13733 log.cc:824] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:04.730893 13736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:04.728016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35691 (local address 127.0.0.1:40265)
0806 20:43:04.728204 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:04.728211 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:04.728513 (+ 302us) server_negotiation.cc:407] Connection header received
0806 20:43:04.728739 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:04.728746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:04.728818 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:04.728952 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:04.730000 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:04.730006 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:04.730012 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:04.730034 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:04.730087 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:04.730109 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:04.730199 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:04.730480 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 20:43:04.730555 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 20:43:04.732844 13733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff: No bootstrap required, opened a new log
I20260806 20:43:04.740424 13733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf92c6f55b4403595e273b7e9f05eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40265 } }
I20260806 20:43:04.740790 13733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:04.740897 13733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf92c6f55b4403595e273b7e9f05eff, State: Initialized, Role: FOLLOWER
I20260806 20:43:04.741451 13733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf92c6f55b4403595e273b7e9f05eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40265 } }
I20260806 20:43:04.741688 13733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:04.741832 13733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:04.742014 13733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:04.744191 13733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf92c6f55b4403595e273b7e9f05eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40265 } }
I20260806 20:43:04.744668 13733 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf92c6f55b4403595e273b7e9f05eff; no voters:
I20260806 20:43:04.745306 13733 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:04.745458 13743 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:04.746145 13743 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [term 1 LEADER]: Becoming Leader. State: Replica: 5cf92c6f55b4403595e273b7e9f05eff, State: Running, Role: LEADER
I20260806 20:43:04.746826 13743 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf92c6f55b4403595e273b7e9f05eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40265 } }
I20260806 20:43:04.747501 13733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:04.749500 13745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf92c6f55b4403595e273b7e9f05eff. Latest consensus state: current_term: 1 leader_uuid: "5cf92c6f55b4403595e273b7e9f05eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf92c6f55b4403595e273b7e9f05eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40265 } } }
I20260806 20:43:04.749788 13744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf92c6f55b4403595e273b7e9f05eff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf92c6f55b4403595e273b7e9f05eff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40265 } } }
I20260806 20:43:04.750078 13745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:04.750102 13744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:04.752020 13750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:04.757333 13750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:04.766803 13750 catalog_manager.cc:1347] Generated new cluster ID: 55b2a9aab2644beaa1234a93bcc2c066
I20260806 20:43:04.766932 13750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:04.784654 13750 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:04.785130 13750 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:04.799152 13750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf92c6f55b4403595e273b7e9f05eff: Generated new TSK 0
I20260806 20:43:04.800037 13750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:04.809041 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13665
W20260806 20:43:04.810325 13727 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50147
W20260806 20:43:04.811124 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13665: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:04.811283 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13665
2026-08-06T20:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1938 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:04 dist-test-slave-tm1g krb5kdc[13764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:04 dist-test-slave-tm1g krb5kdc[13764](info): set up 2 sockets
Aug 06 20:43:04 dist-test-slave-tm1g krb5kdc[13764](info): commencing operation
krb5kdc: starting...
W20260806 20:43:05.934422 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:06 dist-test-slave-tm1g krb5kdc[13764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:06.197340 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:06.530786 13778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:06.531109 13778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:06.531282 13778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:06.538343 13778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:06.538501 13778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:06.538604 13778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:06.538700 13778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:06.538784 13778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:06.548441 13778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:06.549815 13778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:06.551153 13778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:06.562973 13784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:06.563097 13783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:06.563304 13786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:06.564051 13778 server_base.cc:1034] running on GCE node
I20260806 20:43:06.564831 13778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:06.566210 13778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:06.567432 13778 hybrid_clock.cc:648] HybridClock initialized: now 1786048986567398 us; error 71 us; skew 500 ppm
I20260806 20:43:06.567770 13778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:06 dist-test-slave-tm1g krb5kdc[13764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048986, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:06.570981 13778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:06.572399 13778 webserver.cc:460] Webserver started at http://127.0.0.1:40675/ using document root <none> and password file <none>
I20260806 20:43:06.573099 13778 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:06.573247 13778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:06.573560 13778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:06.576602 13778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "de312fd367b04c5385e9188a1c10a0e3"
format_stamp: "Formatted at 2026-08-06 20:43:06 on dist-test-slave-tm1g"
server_key: "8207f4d7041e5b463bbf230a704b63d9"
server_key_iv: "fa23e956ef0bdac587b1b83a43ab9224"
server_key_version: "encryptionkey@0"
I20260806 20:43:06.577389 13778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "de312fd367b04c5385e9188a1c10a0e3"
format_stamp: "Formatted at 2026-08-06 20:43:06 on dist-test-slave-tm1g"
server_key: "8207f4d7041e5b463bbf230a704b63d9"
server_key_iv: "fa23e956ef0bdac587b1b83a43ab9224"
server_key_version: "encryptionkey@0"
I20260806 20:43:06.582546 13778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260806 20:43:06.585891 13793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:06.587181 13778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:43:06.587374 13778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "de312fd367b04c5385e9188a1c10a0e3"
format_stamp: "Formatted at 2026-08-06 20:43:06 on dist-test-slave-tm1g"
server_key: "8207f4d7041e5b463bbf230a704b63d9"
server_key_iv: "fa23e956ef0bdac587b1b83a43ab9224"
server_key_version: "encryptionkey@0"
I20260806 20:43:06.587572 13778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:06.601485 13778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:06.605913 13778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:06.606209 13778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:06.624751 13778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41679
I20260806 20:43:06.624788 13845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41679 every 8 connection(s)
I20260806 20:43:06.626122 13778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:06.629892 13846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:06.633819 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13778
I20260806 20:43:06.634011 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:06.634533 24352 external_mini_cluster.cc:1428] Setting key a82ddefd2e34716c119509205a6149f3
I20260806 20:43:06.640695 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3: Bootstrap starting.
I20260806 20:43:06.643728 13846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:06.644718 13846 log.cc:824] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:06.648134 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3: No bootstrap required, opened a new log
Aug 06 20:43:06 dist-test-slave-tm1g krb5kdc[13764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048986, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 20:43:06.655920 13846 raft_consensus.cc:348] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de312fd367b04c5385e9188a1c10a0e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260806 20:43:06.656296 13846 raft_consensus.cc:374] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:06.656436 13846 raft_consensus.cc:729] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de312fd367b04c5385e9188a1c10a0e3, State: Initialized, Role: FOLLOWER
I20260806 20:43:06.657076 13846 consensus_queue.cc:260] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de312fd367b04c5385e9188a1c10a0e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260806 20:43:06.657337 13846 raft_consensus.cc:388] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:06.657502 13846 raft_consensus.cc:482] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:06.657691 13846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:06.658435 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:06.636304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:48277 (local address 127.0.0.1:41679)
0806 20:43:06.637052 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:06.637063 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:06.637102 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:43:06.639448 (+ 2346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:06.639472 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:06.639841 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:06.640267 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:06.641819 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:06.643200 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:06.644530 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:06.645424 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:06.645613 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:06.650616 (+ 5003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:06.650624 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:06.650637 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:06.650675 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:06.653843 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:06.654742 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:06.654748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:06.654753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:06.654926 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:06.655402 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:06.655412 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:06.655414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:06.655916 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:06.656254 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:06.656731 (+ 477us) server_negotiation.cc:299] Negotiation successful
0806 20:43:06.657230 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":326,"threads_started":1}
I20260806 20:43:06.660710 13846 raft_consensus.cc:504] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de312fd367b04c5385e9188a1c10a0e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260806 20:43:06.661286 13846 leader_election.cc:302] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de312fd367b04c5385e9188a1c10a0e3; no voters:
I20260806 20:43:06.661998 13846 leader_election.cc:288] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:06.662161 13851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:06.662834 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:06.664062 13846 sys_catalog.cc:564] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:06.668737 13851 raft_consensus.cc:686] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [term 1 LEADER]: Becoming Leader. State: Replica: de312fd367b04c5385e9188a1c10a0e3, State: Running, Role: LEADER
I20260806 20:43:06.669538 13851 consensus_queue.cc:237] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de312fd367b04c5385e9188a1c10a0e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } }
I20260806 20:43:06.673697 13852 sys_catalog.cc:455] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de312fd367b04c5385e9188a1c10a0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de312fd367b04c5385e9188a1c10a0e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260806 20:43:06.674149 13852 sys_catalog.cc:458] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:06.674557 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de312fd367b04c5385e9188a1c10a0e3. Latest consensus state: current_term: 1 leader_uuid: "de312fd367b04c5385e9188a1c10a0e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de312fd367b04c5385e9188a1c10a0e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41679 } } }
I20260806 20:43:06.674866 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:06.676617 13872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:06.681631 13872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:06.687443 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:06.669331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:57165 (local address 127.0.0.1:41679)
0806 20:43:06.669474 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:06.669481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:06.669746 (+ 265us) server_negotiation.cc:407] Connection header received
0806 20:43:06.670042 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:06.670053 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:06.670148 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:06.670235 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:06.672351 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:06.673739 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:06.675649 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:06.676628 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:06.676766 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:06.680062 (+ 3296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:06.680071 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:06.680078 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:06.680136 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:06.684141 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:06.685097 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:06.685102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:06.685105 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:06.685255 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:06.685767 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:06.685772 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:06.685775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:06.686143 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:06.686468 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:06.686877 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 20:43:06.687043 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260806 20:43:06.692668 13872 catalog_manager.cc:1347] Generated new cluster ID: fa9b7bc62fc44bc7aa71156b13cb5cc3
I20260806 20:43:06.692788 13872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:06.710719 13872 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:06.712565 13872 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:06.723958 13872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de312fd367b04c5385e9188a1c10a0e3: Generated new TSK 0
I20260806 20:43:06.724781 13872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:06.730870 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13778
I20260806 20:43:06.732249 13841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:06.838402 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13778
W20260806 20:43:06.853798 13762 connection.cc:535] client connection to 127.0.0.1:41679 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41679): Connection reset by peer (error 104)
2026-08-06T20:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2024 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:06 dist-test-slave-tm1g krb5kdc[13881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:06 dist-test-slave-tm1g krb5kdc[13881](info): set up 2 sockets
Aug 06 20:43:06 dist-test-slave-tm1g krb5kdc[13881](info): commencing operation
krb5kdc: starting...
W20260806 20:43:07.958285 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:08 dist-test-slave-tm1g krb5kdc[13881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048988, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:08.207736 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:08.537288 13895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:08.537627 13895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:08.537799 13895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:08.544849 13895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:08.544998 13895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:08.545105 13895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:08.545217 13895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:08.545323 13895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:08.554817 13895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:08.556084 13895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:08.557394 13895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:08.569010 13900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:08.568967 13901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:08.569052 13903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:08.569790 13895 server_base.cc:1034] running on GCE node
I20260806 20:43:08.570647 13895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:08.572038 13895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:08.573277 13895 hybrid_clock.cc:648] HybridClock initialized: now 1786048988573242 us; error 49 us; skew 500 ppm
I20260806 20:43:08.573624 13895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:08 dist-test-slave-tm1g krb5kdc[13881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:08.576659 13895 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:08.578118 13895 webserver.cc:460] Webserver started at http://127.0.0.1:35215/ using document root <none> and password file <none>
I20260806 20:43:08.578840 13895 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:08.578991 13895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:08.579298 13895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:08.582634 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "042c70c94b1a48a893a6b7dfd800957b"
format_stamp: "Formatted at 2026-08-06 20:43:08 on dist-test-slave-tm1g"
server_key: "873a11f82a1b6f79a48da9f7d59c4259"
server_key_iv: "e4bfd4fb82df586f5ee728f3cb49141c"
server_key_version: "encryptionkey@0"
I20260806 20:43:08.583406 13895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "042c70c94b1a48a893a6b7dfd800957b"
format_stamp: "Formatted at 2026-08-06 20:43:08 on dist-test-slave-tm1g"
server_key: "873a11f82a1b6f79a48da9f7d59c4259"
server_key_iv: "e4bfd4fb82df586f5ee728f3cb49141c"
server_key_version: "encryptionkey@0"
I20260806 20:43:08.588344 13895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:43:08.591662 13910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:08.592914 13895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:43:08.593096 13895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "042c70c94b1a48a893a6b7dfd800957b"
format_stamp: "Formatted at 2026-08-06 20:43:08 on dist-test-slave-tm1g"
server_key: "873a11f82a1b6f79a48da9f7d59c4259"
server_key_iv: "e4bfd4fb82df586f5ee728f3cb49141c"
server_key_version: "encryptionkey@0"
I20260806 20:43:08.593287 13895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:08.624155 13895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:08.628129 13895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:08.628420 13895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:08.647091 13895 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40977
I20260806 20:43:08.647130 13963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40977 every 8 connection(s)
I20260806 20:43:08.648435 13895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:08.651636 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 13895
I20260806 20:43:08.651811 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:08.652361 13964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:08.652393 24352 external_mini_cluster.cc:1428] Setting key ad103bd2003145538ea783ddffb66873
I20260806 20:43:08.664711 13964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b: Bootstrap starting.
I20260806 20:43:08.665493 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:08.654225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:42235 (local address 127.0.0.1:40977)
0806 20:43:08.654940 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:08.654954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:08.654998 (+ 44us) server_negotiation.cc:407] Connection header received
0806 20:43:08.657013 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:08.657034 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:08.657396 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:08.657785 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:08.658374 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:08.659711 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:08.661019 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:08.661720 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:08.661922 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:08.663213 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:08.663220 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:08.663232 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:08.663729 (+ 497us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:08.663802 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:08.663831 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:08.664000 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:08.664281 (+ 281us) server_negotiation.cc:299] Negotiation successful
0806 20:43:08.664439 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":303,"threads_started":1}
I20260806 20:43:08.667940 13964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:08.669083 13964 log.cc:824] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:08.669363 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:08.671826 13964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b: No bootstrap required, opened a new log
I20260806 20:43:08.680892 13967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:08.673813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:48467 (local address 127.0.0.1:40977)
0806 20:43:08.674054 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:08.674063 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:08.674476 (+ 413us) server_negotiation.cc:407] Connection header received
0806 20:43:08.674659 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:08.674662 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:08.674730 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:08.674804 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:08.675483 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:08.676272 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:08.678238 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:08.678774 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:08.678882 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:08.680089 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:08.680094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:08.680097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:08.680119 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:08.680158 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:08.680175 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:08.680263 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:08.680483 (+ 220us) server_negotiation.cc:299] Negotiation successful
0806 20:43:08.680550 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 20:43:08.681797 13964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042c70c94b1a48a893a6b7dfd800957b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260806 20:43:08.682200 13964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:08.682348 13964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 042c70c94b1a48a893a6b7dfd800957b, State: Initialized, Role: FOLLOWER
I20260806 20:43:08.682957 13964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042c70c94b1a48a893a6b7dfd800957b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260806 20:43:08.683212 13964 raft_consensus.cc:388] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:08.683353 13964 raft_consensus.cc:482] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:08.683537 13964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:08.685706 13964 raft_consensus.cc:504] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042c70c94b1a48a893a6b7dfd800957b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260806 20:43:08.686211 13964 leader_election.cc:302] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 042c70c94b1a48a893a6b7dfd800957b; no voters:
I20260806 20:43:08.686841 13964 leader_election.cc:288] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:08.687005 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:08.687690 13974 raft_consensus.cc:686] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [term 1 LEADER]: Becoming Leader. State: Replica: 042c70c94b1a48a893a6b7dfd800957b, State: Running, Role: LEADER
I20260806 20:43:08.688235 13974 consensus_queue.cc:237] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042c70c94b1a48a893a6b7dfd800957b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } }
I20260806 20:43:08.688946 13964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:08.691030 13975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "042c70c94b1a48a893a6b7dfd800957b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042c70c94b1a48a893a6b7dfd800957b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260806 20:43:08.691082 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 042c70c94b1a48a893a6b7dfd800957b. Latest consensus state: current_term: 1 leader_uuid: "042c70c94b1a48a893a6b7dfd800957b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "042c70c94b1a48a893a6b7dfd800957b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40977 } } }
I20260806 20:43:08.691505 13975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:08.691510 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:08.699568 13981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:08.704663 13981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:08.712457 13981 catalog_manager.cc:1347] Generated new cluster ID: 67c880f489a8477fb0bfbe42a409af29
I20260806 20:43:08.712563 13981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:08.725746 13981 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:08.727203 13981 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:08.734541 13981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 042c70c94b1a48a893a6b7dfd800957b: Generated new TSK 0
I20260806 20:43:08.735332 13981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:08.743768 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13895
W20260806 20:43:08.744856 13959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:42235
W20260806 20:43:08.745581 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 13895: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:08.745733 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 13895
W20260806 20:43:08.763738 13878 connection.cc:535] client connection to 127.0.0.1:40977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40977): Connection reset by peer (error 104)
2026-08-06T20:43:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1909 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:08 dist-test-slave-tm1g krb5kdc[13995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:08 dist-test-slave-tm1g krb5kdc[13995](info): set up 2 sockets
Aug 06 20:43:08 dist-test-slave-tm1g krb5kdc[13995](info): commencing operation
krb5kdc: starting...
W20260806 20:43:09.861861 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:10 dist-test-slave-tm1g krb5kdc[13995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048990, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:10.104835 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40893
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:10.433588 14008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:10.433853 14008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:10.433943 14008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:10.440881 14008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:10.440984 14008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:10.441030 14008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:10.441092 14008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:10.441135 14008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:10.450457 14008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:10.451597 14008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:10.452847 14008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:10.464068 14016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:10.464206 14014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:10.464473 14013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:10.465232 14008 server_base.cc:1034] running on GCE node
I20260806 20:43:10.466033 14008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:10.467371 14008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:10.468593 14008 hybrid_clock.cc:648] HybridClock initialized: now 1786048990468551 us; error 38 us; skew 500 ppm
I20260806 20:43:10.468933 14008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:10 dist-test-slave-tm1g krb5kdc[13995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048990, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:10.477103 14008 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:10.478643 14008 webserver.cc:460] Webserver started at http://127.0.0.1:33519/ using document root <none> and password file <none>
I20260806 20:43:10.479316 14008 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:10.479458 14008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:10.479776 14008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:10.482858 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "7590f538b0b145d7abe07e4c72b97d90"
format_stamp: "Formatted at 2026-08-06 20:43:10 on dist-test-slave-tm1g"
server_key: "adc9ddffd2c659c2e1713eb3c58ee3b3"
server_key_iv: "5ed4ac5ff305195188aac10e70618b4a"
server_key_version: "encryptionkey@0"
I20260806 20:43:10.483647 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "7590f538b0b145d7abe07e4c72b97d90"
format_stamp: "Formatted at 2026-08-06 20:43:10 on dist-test-slave-tm1g"
server_key: "adc9ddffd2c659c2e1713eb3c58ee3b3"
server_key_iv: "5ed4ac5ff305195188aac10e70618b4a"
server_key_version: "encryptionkey@0"
I20260806 20:43:10.488595 14008 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 20:43:10.491906 14023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:10.493119 14008 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260806 20:43:10.493290 14008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "7590f538b0b145d7abe07e4c72b97d90"
format_stamp: "Formatted at 2026-08-06 20:43:10 on dist-test-slave-tm1g"
server_key: "adc9ddffd2c659c2e1713eb3c58ee3b3"
server_key_iv: "5ed4ac5ff305195188aac10e70618b4a"
server_key_version: "encryptionkey@0"
I20260806 20:43:10.493485 14008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:10.499130 14008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:10.502789 14008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:10.503090 14008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:10.521399 14008 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40893
I20260806 20:43:10.521471 14077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40893 every 8 connection(s)
I20260806 20:43:10.522773 14008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:10.526238 14078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:10.527729 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14008
I20260806 20:43:10.527897 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:10.528338 24352 external_mini_cluster.cc:1428] Setting key 87e3f7d5f8ec73e8cb5b1499efa4c999
I20260806 20:43:10.536366 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:10.530093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:57347 (local address 127.0.0.1:40893)
0806 20:43:10.530969 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:10.530984 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:10.531038 (+ 54us) server_negotiation.cc:407] Connection header received
0806 20:43:10.532745 (+ 1707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:10.532772 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:10.533198 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:10.533617 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:10.534570 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:10.534579 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:10.534596 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:10.534613 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:10.534672 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:10.534694 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:10.534904 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:10.535055 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 20:43:10.535231 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":352,"threads_started":1}
I20260806 20:43:10.539755 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:10.540284 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90: Bootstrap starting.
I20260806 20:43:10.544175 14078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:10.545506 14078 log.cc:824] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:10.547688 14081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:10.544993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:38719 (local address 127.0.0.1:40893)
0806 20:43:10.545190 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:10.545199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:10.545714 (+ 515us) server_negotiation.cc:407] Connection header received
0806 20:43:10.545841 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:10.545845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:10.545887 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:10.545976 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:10.547018 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:10.547021 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:10.547024 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:10.547038 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:10.547072 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:10.547085 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:10.547162 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:10.547322 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 20:43:10.547365 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 20:43:10.548616 14078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90: No bootstrap required, opened a new log
I20260806 20:43:10.556038 14078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7590f538b0b145d7abe07e4c72b97d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260806 20:43:10.556381 14078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:10.556491 14078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7590f538b0b145d7abe07e4c72b97d90, State: Initialized, Role: FOLLOWER
I20260806 20:43:10.556999 14078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7590f538b0b145d7abe07e4c72b97d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260806 20:43:10.557224 14078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:10.557379 14078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:10.557528 14078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:10.559677 14078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7590f538b0b145d7abe07e4c72b97d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260806 20:43:10.560142 14078 leader_election.cc:302] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7590f538b0b145d7abe07e4c72b97d90; no voters:
I20260806 20:43:10.560663 14078 leader_election.cc:288] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:10.560797 14088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:10.561448 14088 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [term 1 LEADER]: Becoming Leader. State: Replica: 7590f538b0b145d7abe07e4c72b97d90, State: Running, Role: LEADER
I20260806 20:43:10.561975 14088 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7590f538b0b145d7abe07e4c72b97d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } }
I20260806 20:43:10.562783 14078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:10.564744 14091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7590f538b0b145d7abe07e4c72b97d90. Latest consensus state: current_term: 1 leader_uuid: "7590f538b0b145d7abe07e4c72b97d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7590f538b0b145d7abe07e4c72b97d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } } }
I20260806 20:43:10.565220 14091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:10.565721 14090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7590f538b0b145d7abe07e4c72b97d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7590f538b0b145d7abe07e4c72b97d90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40893 } } }
I20260806 20:43:10.566013 14090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:10.567247 14097 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:10.571424 14097 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:10.580654 14097 catalog_manager.cc:1347] Generated new cluster ID: f13376ee0f14427eb3c22aa4fd4615f0
I20260806 20:43:10.580775 14097 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:10.598026 14097 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:10.598465 14097 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:10.605552 14097 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7590f538b0b145d7abe07e4c72b97d90: Generated new TSK 0
I20260806 20:43:10.606232 14097 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:10.613282 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14008
W20260806 20:43:10.614396 14073 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:57347
W20260806 20:43:10.615185 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14008: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:10.615474 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14008
2026-08-06T20:43:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1869 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:10 dist-test-slave-tm1g krb5kdc[14110](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:10 dist-test-slave-tm1g krb5kdc[14110](info): set up 2 sockets
Aug 06 20:43:10 dist-test-slave-tm1g krb5kdc[14110](info): commencing operation
krb5kdc: starting...
W20260806 20:43:11.733800 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:11 dist-test-slave-tm1g krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:11.981415 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46341
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46341
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:12.306605 14123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:12.306912 14123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:12.307066 14123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:12.314074 14123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:12.314206 14123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:12.314306 14123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:12.314441 14123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:12.314514 14123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:12.323985 14123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46341
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:12.325224 14123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:12.326613 14123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:12.338168 14128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:12.338145 14129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:12.338119 14131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:12.339083 14123 server_base.cc:1034] running on GCE node
I20260806 20:43:12.339771 14123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:12.341045 14123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:12.342264 14123 hybrid_clock.cc:648] HybridClock initialized: now 1786048992342217 us; error 57 us; skew 500 ppm
I20260806 20:43:12.342623 14123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:12 dist-test-slave-tm1g krb5kdc[14110](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048992, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:12.345656 14123 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:12.347010 14123 webserver.cc:460] Webserver started at http://127.0.0.1:33931/ using document root <none> and password file <none>
I20260806 20:43:12.347613 14123 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:12.347710 14123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:12.347956 14123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:12.350768 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "f2c7b0d0efe448369c3457e89287df0f"
format_stamp: "Formatted at 2026-08-06 20:43:12 on dist-test-slave-tm1g"
server_key: "1c8cebd8541f11c33ce440c77b8b95cb"
server_key_iv: "c65060b0f7938ac390f753a854a646f3"
server_key_version: "encryptionkey@0"
I20260806 20:43:12.351462 14123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "f2c7b0d0efe448369c3457e89287df0f"
format_stamp: "Formatted at 2026-08-06 20:43:12 on dist-test-slave-tm1g"
server_key: "1c8cebd8541f11c33ce440c77b8b95cb"
server_key_iv: "c65060b0f7938ac390f753a854a646f3"
server_key_version: "encryptionkey@0"
I20260806 20:43:12.356238 14123 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 20:43:12.359431 14138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:12.360626 14123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260806 20:43:12.360760 14123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "f2c7b0d0efe448369c3457e89287df0f"
format_stamp: "Formatted at 2026-08-06 20:43:12 on dist-test-slave-tm1g"
server_key: "1c8cebd8541f11c33ce440c77b8b95cb"
server_key_iv: "c65060b0f7938ac390f753a854a646f3"
server_key_version: "encryptionkey@0"
I20260806 20:43:12.360925 14123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:12.376690 14123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:12.380668 14123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:12.380903 14123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:12.398830 14123 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46341
I20260806 20:43:12.398874 14191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46341 every 8 connection(s)
I20260806 20:43:12.400110 14123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:12.403754 14192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:12.403729 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14123
I20260806 20:43:12.403930 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:12.404516 24352 external_mini_cluster.cc:1428] Setting key 36a6c1f27e353be916ce6aed51a1bfe1
I20260806 20:43:12.418164 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f: Bootstrap starting.
Aug 06 20:43:12 dist-test-slave-tm1g krb5kdc[14110](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048991, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 20:43:12.421442 14192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:12.422570 14192 log.cc:824] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:12.425220 14192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f: No bootstrap required, opened a new log
I20260806 20:43:12.426939 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:12.406227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:34999 (local address 127.0.0.1:46341)
0806 20:43:12.407082 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:12.407093 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:12.407133 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:43:12.408769 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:12.408790 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:12.409151 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:12.409471 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:12.411350 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:12.412578 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:12.414459 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:12.415136 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:12.415264 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:12.419914 (+ 4650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:12.419924 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:12.419936 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:12.419975 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:12.423021 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:12.423754 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:12.423760 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:12.423765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:12.423916 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:12.424364 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:12.424368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:12.424370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:12.424846 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:12.425190 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:12.425688 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 20:43:12.425987 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":383,"threads_started":1}
I20260806 20:43:12.430485 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:12.433079 14192 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7b0d0efe448369c3457e89287df0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260806 20:43:12.433511 14192 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:12.433624 14192 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2c7b0d0efe448369c3457e89287df0f, State: Initialized, Role: FOLLOWER
I20260806 20:43:12.434142 14192 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7b0d0efe448369c3457e89287df0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260806 20:43:12.434428 14192 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:12.434595 14192 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:12.434826 14192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:12.437132 14192 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7b0d0efe448369c3457e89287df0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260806 20:43:12.437636 14192 leader_election.cc:302] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2c7b0d0efe448369c3457e89287df0f; no voters:
I20260806 20:43:12.438359 14192 leader_election.cc:288] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:12.438436 14202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:12.440893 14192 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:12.442612 14202 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [term 1 LEADER]: Becoming Leader. State: Replica: f2c7b0d0efe448369c3457e89287df0f, State: Running, Role: LEADER
I20260806 20:43:12.443432 14202 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7b0d0efe448369c3457e89287df0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } }
I20260806 20:43:12.446941 14204 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2c7b0d0efe448369c3457e89287df0f. Latest consensus state: current_term: 1 leader_uuid: "f2c7b0d0efe448369c3457e89287df0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7b0d0efe448369c3457e89287df0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } } }
I20260806 20:43:12.447345 14204 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:12.447690 14203 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2c7b0d0efe448369c3457e89287df0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2c7b0d0efe448369c3457e89287df0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46341 } } }
I20260806 20:43:12.447980 14203 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:12.450527 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:12.434849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:46353 (local address 127.0.0.1:46341)
0806 20:43:12.435024 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:12.435032 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:12.435417 (+ 385us) server_negotiation.cc:407] Connection header received
0806 20:43:12.435601 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:12.435606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:12.435680 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:12.435796 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:12.437132 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:12.437975 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:12.439737 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:12.440185 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:12.440316 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:12.442860 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:12.442866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:12.442870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:12.442904 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:12.446451 (+ 3547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:12.447462 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:12.447468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:12.447471 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:12.447633 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:12.448268 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:12.448273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:12.448276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:12.448716 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:12.449502 (+ 786us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:12.449802 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 20:43:12.449985 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260806 20:43:12.453135 14212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:12.457413 14212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:12.465978 14212 catalog_manager.cc:1347] Generated new cluster ID: eb0f5282a85b4e1390e98dfc9584ec0c
I20260806 20:43:12.466257 14212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:12.493202 14212 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:12.494980 14212 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:12.502166 14212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2c7b0d0efe448369c3457e89287df0f: Generated new TSK 0
I20260806 20:43:12.503118 14212 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:12.516002 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14123
I20260806 20:43:12.517196 14187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:12.607992 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14123
W20260806 20:43:12.626097 14108 connection.cc:535] client connection to 127.0.0.1:46341 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46341): Connection reset by peer (error 104)
2026-08-06T20:43:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:12 dist-test-slave-tm1g krb5kdc[14224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:12 dist-test-slave-tm1g krb5kdc[14224](info): set up 2 sockets
Aug 06 20:43:12 dist-test-slave-tm1g krb5kdc[14224](info): commencing operation
krb5kdc: starting...
W20260806 20:43:13.743413 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:13 dist-test-slave-tm1g krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:13.995173 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42617
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:14.325081 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:14.325382 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:14.325564 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:14.332743 14238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:14.332866 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:14.332970 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:14.333069 14238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:14.333151 14238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:14.342851 14238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42617
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42819
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:14.344174 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:14.345561 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:14.357017 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:14.357046 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:14.357132 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:14.358052 14238 server_base.cc:1034] running on GCE node
I20260806 20:43:14.358791 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:14.360052 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:14.361274 14238 hybrid_clock.cc:648] HybridClock initialized: now 1786048994361223 us; error 39 us; skew 500 ppm
I20260806 20:43:14.361644 14238 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:14 dist-test-slave-tm1g krb5kdc[14224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048994, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:14.364741 14238 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:14.366189 14238 webserver.cc:460] Webserver started at http://127.0.0.1:45413/ using document root <none> and password file <none>
I20260806 20:43:14.366904 14238 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:14.367048 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:14.367352 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:14.370355 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9"
format_stamp: "Formatted at 2026-08-06 20:43:14 on dist-test-slave-tm1g"
server_key: "a3ce0071e1783f244a60370988938e8b"
server_key_iv: "64540a5dbbbb2421773ae21cd5fa9bb2"
server_key_version: "encryptionkey@0"
I20260806 20:43:14.371132 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9"
format_stamp: "Formatted at 2026-08-06 20:43:14 on dist-test-slave-tm1g"
server_key: "a3ce0071e1783f244a60370988938e8b"
server_key_iv: "64540a5dbbbb2421773ae21cd5fa9bb2"
server_key_version: "encryptionkey@0"
I20260806 20:43:14.376060 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260806 20:43:14.379537 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:14.380749 14238 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:43:14.380934 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9"
format_stamp: "Formatted at 2026-08-06 20:43:14 on dist-test-slave-tm1g"
server_key: "a3ce0071e1783f244a60370988938e8b"
server_key_iv: "64540a5dbbbb2421773ae21cd5fa9bb2"
server_key_version: "encryptionkey@0"
I20260806 20:43:14.381127 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:14.418165 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:14.422251 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:14.422573 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:14.440991 14238 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42819
I20260806 20:43:14.441033 14304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42819 every 8 connection(s)
I20260806 20:43:14.442328 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:14.445825 14305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:14.449193 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14238
I20260806 20:43:14.449379 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:14.449819 24352 external_mini_cluster.cc:1428] Setting key 89e42a5bcb52150e604a1d23a2b9a4a1
I20260806 20:43:14.456135 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9: Bootstrap starting.
I20260806 20:43:14.459327 14305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:14.460574 14305 log.cc:824] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:14.463238 14305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9: No bootstrap required, opened a new log
I20260806 20:43:14.467327 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:14.456091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:37109 (local address 127.0.0.1:42819)
0806 20:43:14.456882 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:14.456893 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:14.456935 (+ 42us) server_negotiation.cc:407] Connection header received
0806 20:43:14.458754 (+ 1819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:14.458775 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:14.459129 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:14.459462 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:14.460097 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:14.461508 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:14.462912 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:14.463626 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:14.463819 (+ 193us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:14.465061 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:14.465068 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:14.465080 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:14.465645 (+ 565us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:14.465719 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:14.465750 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:14.465922 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:14.466206 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 20:43:14.466405 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":383,"threads_started":1}
I20260806 20:43:14.470984 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:14.471225 14305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42819 } }
I20260806 20:43:14.471567 14305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:14.471670 14305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5fbae6fb00445c9ec5d60d9e0972e9, State: Initialized, Role: FOLLOWER
I20260806 20:43:14.472172 14305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42819 } }
I20260806 20:43:14.472396 14305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:14.472527 14305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:14.472694 14305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:14.475069 14305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42819 } }
I20260806 20:43:14.475560 14305 leader_election.cc:302] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5fbae6fb00445c9ec5d60d9e0972e9; no voters:
I20260806 20:43:14.476212 14305 leader_election.cc:288] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:14.476303 14315 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:14.478783 14305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:14.482689 14308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:14.475552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:36951 (local address 127.0.0.1:42819)
0806 20:43:14.475745 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:14.475751 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:14.476223 (+ 472us) server_negotiation.cc:407] Connection header received
0806 20:43:14.476455 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:14.476460 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:14.476554 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:14.476683 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:14.477473 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:14.478256 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:14.479864 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:14.480358 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:14.480525 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:14.481880 (+ 1355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:14.481885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:14.481888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:14.481904 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:14.481947 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:14.481962 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:14.482081 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:14.482272 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 20:43:14.482356 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260806 20:43:14.483356 14315 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5fbae6fb00445c9ec5d60d9e0972e9, State: Running, Role: LEADER
I20260806 20:43:14.484061 14315 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42819 } }
I20260806 20:43:14.487761 14318 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5fbae6fb00445c9ec5d60d9e0972e9. Latest consensus state: current_term: 1 leader_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42819 } } }
I20260806 20:43:14.488169 14318 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:14.488657 14317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5fbae6fb00445c9ec5d60d9e0972e9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42819 } } }
I20260806 20:43:14.489025 14317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:14.490900 14327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:14.495460 14327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:14.503468 14327 catalog_manager.cc:1347] Generated new cluster ID: 276b84b80f244109bf0a1ef729744d3c
I20260806 20:43:14.503577 14327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:14.534597 14327 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:14.536415 14327 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:14.552959 14327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d5fbae6fb00445c9ec5d60d9e0972e9: Generated new TSK 0
I20260806 20:43:14.553725 14327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:14.563053 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14238
W20260806 20:43:14.564056 14300 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:37109
W20260806 20:43:14.564765 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14238: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:14.564914 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14238
W20260806 20:43:14.583069 14222 connection.cc:535] client connection to 127.0.0.1:42819 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42819): Connection reset by peer (error 104)
2026-08-06T20:43:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1956 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:14 dist-test-slave-tm1g krb5kdc[14337](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:14 dist-test-slave-tm1g krb5kdc[14337](info): set up 2 sockets
Aug 06 20:43:14 dist-test-slave-tm1g krb5kdc[14337](info): commencing operation
krb5kdc: starting...
W20260806 20:43:15.674167 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:15 dist-test-slave-tm1g krb5kdc[14337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 20:43:15.923524 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:16.249259 14350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:16.249544 14350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:16.249697 14350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:16.256822 14350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:16.256916 14350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:16.257032 14350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:16.257129 14350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:16.257206 14350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:16.266731 14350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45973
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:16.267911 14350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:16.269204 14350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:16.280753 14358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:16.280848 14355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:16.280825 14356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:16.281629 14350 server_base.cc:1034] running on GCE node
I20260806 20:43:16.282351 14350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:16.283705 14350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:16.284924 14350 hybrid_clock.cc:648] HybridClock initialized: now 1786048996284877 us; error 31 us; skew 500 ppm
I20260806 20:43:16.285266 14350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:16 dist-test-slave-tm1g krb5kdc[14337](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:16.291757 14350 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 20:43:16.293213 14350 webserver.cc:460] Webserver started at http://127.0.0.1:45799/ using document root <none> and password file <none>
I20260806 20:43:16.293892 14350 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:16.294023 14350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:16.294396 14350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:16.297350 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "5ed25fcc67d24ec0b4b52103079f3164"
format_stamp: "Formatted at 2026-08-06 20:43:16 on dist-test-slave-tm1g"
server_key: "25be038948213e441f189de6c515db94"
server_key_iv: "5e85027141d90349cb8b54d6cccdbeb3"
server_key_version: "encryptionkey@0"
I20260806 20:43:16.298112 14350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "5ed25fcc67d24ec0b4b52103079f3164"
format_stamp: "Formatted at 2026-08-06 20:43:16 on dist-test-slave-tm1g"
server_key: "25be038948213e441f189de6c515db94"
server_key_iv: "5e85027141d90349cb8b54d6cccdbeb3"
server_key_version: "encryptionkey@0"
I20260806 20:43:16.304387 14350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 20:43:16.307822 14365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:16.309062 14350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:43:16.309229 14350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "5ed25fcc67d24ec0b4b52103079f3164"
format_stamp: "Formatted at 2026-08-06 20:43:16 on dist-test-slave-tm1g"
server_key: "25be038948213e441f189de6c515db94"
server_key_iv: "5e85027141d90349cb8b54d6cccdbeb3"
server_key_version: "encryptionkey@0"
I20260806 20:43:16.309415 14350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:16.315021 14350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:16.318940 14350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:16.319216 14350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:16.337730 14350 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45973
I20260806 20:43:16.337809 14417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45973 every 8 connection(s)
I20260806 20:43:16.339228 14350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:16.342932 14418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:16.347105 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14350
I20260806 20:43:16.347296 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:16.347865 24352 external_mini_cluster.cc:1428] Setting key 0f9429a3620b146e3532b7ccef3ff1be
I20260806 20:43:16.355183 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:16.349564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:37849 (local address 127.0.0.1:45973)
0806 20:43:16.350293 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:16.350351 (+ 58us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:16.350396 (+ 45us) server_negotiation.cc:407] Connection header received
0806 20:43:16.351957 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:16.351979 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:16.352306 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:16.352618 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:16.353624 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:16.353629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:16.353641 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:16.353657 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:16.353711 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:16.353728 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:16.353896 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:16.354023 (+ 127us) server_negotiation.cc:299] Negotiation successful
0806 20:43:16.354218 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":325,"threads_started":1}
I20260806 20:43:16.356045 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164: Bootstrap starting.
I20260806 20:43:16.358917 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:16.359752 14418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:16.360948 14418 log.cc:824] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:16.363719 14418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164: No bootstrap required, opened a new log
I20260806 20:43:16.366605 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:16.364143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.78:51395 (local address 127.0.0.1:45973)
0806 20:43:16.364312 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:16.364320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:16.364683 (+ 363us) server_negotiation.cc:407] Connection header received
0806 20:43:16.364866 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:16.364871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:16.364927 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:16.365019 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:16.365897 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:16.365902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:16.365907 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:16.365923 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:16.365973 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:16.365987 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:16.366102 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:16.366221 (+ 119us) server_negotiation.cc:299] Negotiation successful
0806 20:43:16.366288 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 20:43:16.371744 14418 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed25fcc67d24ec0b4b52103079f3164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260806 20:43:16.372203 14418 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:16.372306 14418 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ed25fcc67d24ec0b4b52103079f3164, State: Initialized, Role: FOLLOWER
I20260806 20:43:16.372872 14418 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed25fcc67d24ec0b4b52103079f3164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260806 20:43:16.373106 14418 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:16.373247 14418 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:16.373409 14418 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:16.375759 14418 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed25fcc67d24ec0b4b52103079f3164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260806 20:43:16.376199 14418 leader_election.cc:302] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ed25fcc67d24ec0b4b52103079f3164; no voters:
I20260806 20:43:16.376730 14418 leader_election.cc:288] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:16.376866 14428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:16.377514 14428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [term 1 LEADER]: Becoming Leader. State: Replica: 5ed25fcc67d24ec0b4b52103079f3164, State: Running, Role: LEADER
I20260806 20:43:16.378043 14428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed25fcc67d24ec0b4b52103079f3164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } }
I20260806 20:43:16.378693 14418 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:16.381068 14430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ed25fcc67d24ec0b4b52103079f3164. Latest consensus state: current_term: 1 leader_uuid: "5ed25fcc67d24ec0b4b52103079f3164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed25fcc67d24ec0b4b52103079f3164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } } }
I20260806 20:43:16.381547 14430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:16.383250 14437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:16.383293 14429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ed25fcc67d24ec0b4b52103079f3164" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ed25fcc67d24ec0b4b52103079f3164" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45973 } } }
I20260806 20:43:16.383636 14429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:16.388161 14437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:16.396400 14437 catalog_manager.cc:1347] Generated new cluster ID: 67e7fc72625f43558b43f3717d5f44ee
I20260806 20:43:16.396503 14437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:16.417583 14437 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:16.418063 14437 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:16.426847 14437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ed25fcc67d24ec0b4b52103079f3164: Generated new TSK 0
I20260806 20:43:16.427820 14437 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:16.441737 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14350
W20260806 20:43:16.442787 14413 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.78:37849
W20260806 20:43:16.443564 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14350: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:16.443730 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14350
2026-08-06T20:43:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:16 dist-test-slave-tm1g krb5kdc[14449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:16 dist-test-slave-tm1g krb5kdc[14449](info): set up 2 sockets
Aug 06 20:43:16 dist-test-slave-tm1g krb5kdc[14449](info): commencing operation
krb5kdc: starting...
W20260806 20:43:17.558113 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:17 dist-test-slave-tm1g krb5kdc[14449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:17.815449 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45137
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:42069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:45137
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:18.141733 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:18.142052 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:18.142225 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:18.149449 14463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:18.149600 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:18.149705 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:18.149806 14463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:18.149883 14463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:18.159265 14463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:42069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:45137
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:45137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:18.160517 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:18.161839 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:18.173266 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:18.173345 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:18.173311 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:18.174510 14463 server_base.cc:1034] running on GCE node
I20260806 20:43:18.175228 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:18.176555 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:18.177775 14463 hybrid_clock.cc:648] HybridClock initialized: now 1786048998177745 us; error 56 us; skew 500 ppm
I20260806 20:43:18.178112 14463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:18 dist-test-slave-tm1g krb5kdc[14449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:18.181145 14463 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:18.182687 14463 webserver.cc:460] Webserver started at http://127.23.200.62:38009/ using document root <none> and password file <none>
I20260806 20:43:18.183347 14463 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:18.183486 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:18.183779 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:18.186740 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "66c77f71e3d64ac4b6e14abd2283d1db"
format_stamp: "Formatted at 2026-08-06 20:43:18 on dist-test-slave-tm1g"
server_key: "951151b37b416fb7cf7d660452b248e5"
server_key_iv: "83fcde5634a3156a41adf3b425cec6e1"
server_key_version: "encryptionkey@0"
I20260806 20:43:18.187498 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "66c77f71e3d64ac4b6e14abd2283d1db"
format_stamp: "Formatted at 2026-08-06 20:43:18 on dist-test-slave-tm1g"
server_key: "951151b37b416fb7cf7d660452b248e5"
server_key_iv: "83fcde5634a3156a41adf3b425cec6e1"
server_key_version: "encryptionkey@0"
I20260806 20:43:18.192360 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 20:43:18.195736 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:18.196904 14463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:43:18.197083 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "66c77f71e3d64ac4b6e14abd2283d1db"
format_stamp: "Formatted at 2026-08-06 20:43:18 on dist-test-slave-tm1g"
server_key: "951151b37b416fb7cf7d660452b248e5"
server_key_iv: "83fcde5634a3156a41adf3b425cec6e1"
server_key_version: "encryptionkey@0"
I20260806 20:43:18.197278 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:18.219012 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:18.223188 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:18.223476 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:18.241607 14463 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:45137
I20260806 20:43:18.241631 14529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:45137 every 8 connection(s)
I20260806 20:43:18.242954 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:18.246492 14530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:18.249007 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14463
I20260806 20:43:18.249183 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:18.249646 24352 external_mini_cluster.cc:1428] Setting key bf3b7b99516b459de5574c2e789862cf
I20260806 20:43:18.259189 14530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db: Bootstrap starting.
I20260806 20:43:18.262624 14530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:18.263592 14530 log.cc:824] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db: Log is configured to *not* fsync() on all Append() calls
Aug 06 20:43:18 dist-test-slave-tm1g krb5kdc[14449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:18.266717 14530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db: No bootstrap required, opened a new log
I20260806 20:43:18.273480 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:18.251510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32955 (local address 127.23.200.62:45137)
0806 20:43:18.252295 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:18.252311 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:18.252365 (+ 54us) server_negotiation.cc:407] Connection header received
0806 20:43:18.254714 (+ 2349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:18.254741 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:18.255241 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:18.255696 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:18.257064 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:18.259029 (+ 1965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:18.260292 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:18.261385 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:18.261569 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:18.266083 (+ 4514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:18.266091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:18.266104 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:18.266142 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:18.269237 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:18.270149 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:18.270155 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:18.270160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:18.270392 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:18.270912 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:18.270916 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:18.270918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:18.271425 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:18.271683 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:18.272163 (+ 480us) server_negotiation.cc:299] Negotiation successful
0806 20:43:18.272469 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":347,"threads_started":1}
I20260806 20:43:18.275467 14530 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45137 } }
I20260806 20:43:18.275857 14530 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:18.275978 14530 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c77f71e3d64ac4b6e14abd2283d1db, State: Initialized, Role: FOLLOWER
I20260806 20:43:18.276518 14530 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45137 } }
I20260806 20:43:18.276767 14530 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:18.276919 14530 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:18.277100 14530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:18.277213 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:18.280246 14530 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45137 } }
I20260806 20:43:18.280857 14530 leader_election.cc:302] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66c77f71e3d64ac4b6e14abd2283d1db; no voters:
I20260806 20:43:18.281570 14530 leader_election.cc:288] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:18.281687 14538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:18.283998 14530 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:18.287611 14538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [term 1 LEADER]: Becoming Leader. State: Replica: 66c77f71e3d64ac4b6e14abd2283d1db, State: Running, Role: LEADER
I20260806 20:43:18.288283 14538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45137 } }
I20260806 20:43:18.299104 14541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66c77f71e3d64ac4b6e14abd2283d1db. Latest consensus state: current_term: 1 leader_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45137 } } }
I20260806 20:43:18.299574 14541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:18.302345 14540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66c77f71e3d64ac4b6e14abd2283d1db" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 45137 } } }
I20260806 20:43:18.302672 14540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:18.303570 14556 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:18.308251 14533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:18.282749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56683 (local address 127.23.200.62:45137)
0806 20:43:18.282967 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:18.282975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:18.286972 (+ 3997us) server_negotiation.cc:407] Connection header received
0806 20:43:18.287310 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:18.287327 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:18.287434 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:18.287524 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:18.289754 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:18.291395 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:18.293719 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:18.294671 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:18.294825 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:18.298996 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:18.299007 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:18.299015 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:18.299065 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:18.303920 (+ 4855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:18.304823 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:18.304827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:18.304829 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:18.304946 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:18.305438 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:18.305441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:18.305443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:18.305725 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:18.306392 (+ 667us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:18.307661 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0806 20:43:18.307858 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 20:43:18.309602 14556 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:18.317775 14556 catalog_manager.cc:1347] Generated new cluster ID: f556e289dc4146b7bd6ae3718e4a95e7
I20260806 20:43:18.317881 14556 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:18.357175 14556 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:18.358713 14556 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:18.374722 14556 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66c77f71e3d64ac4b6e14abd2283d1db: Generated new TSK 0
I20260806 20:43:18.375484 14556 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:18.387133 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14463
I20260806 20:43:18.388245 14525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:18.480638 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14463
W20260806 20:43:18.498914 14447 connection.cc:535] client connection to 127.23.200.62:45137 recv error: Network error: failed to read from TLS socket (remote: 127.23.200.62:45137): Connection reset by peer (error 104)
2026-08-06T20:43:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2037 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:18 dist-test-slave-tm1g krb5kdc[14564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:18 dist-test-slave-tm1g krb5kdc[14564](info): set up 2 sockets
Aug 06 20:43:18 dist-test-slave-tm1g krb5kdc[14564](info): commencing operation
krb5kdc: starting...
W20260806 20:43:19.598242 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:19 dist-test-slave-tm1g krb5kdc[14564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786048999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:19.849812 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40573
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:44447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:40573
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:20.171599 14578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:20.171945 14578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:20.172115 14578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:20.179138 14578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:20.179284 14578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:20.179401 14578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:20.179514 14578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:20.179596 14578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:20.189129 14578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:44447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:40573
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40573
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:20.190421 14578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:20.191809 14578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:20.203425 14586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:20.203596 14584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:20.203788 14583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:20.204649 14578 server_base.cc:1034] running on GCE node
I20260806 20:43:20.205466 14578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:20.206899 14578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:20.208128 14578 hybrid_clock.cc:648] HybridClock initialized: now 1786049000208088 us; error 50 us; skew 500 ppm
I20260806 20:43:20.208474 14578 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:20 dist-test-slave-tm1g krb5kdc[14564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049000, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:20.211571 14578 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:20.212919 14578 webserver.cc:460] Webserver started at http://127.23.200.62:33573/ using document root <none> and password file <none>
I20260806 20:43:20.213596 14578 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:20.213743 14578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:20.214042 14578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:20.217074 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "f95ba709c0c349b2a92db46aa37b2bea"
format_stamp: "Formatted at 2026-08-06 20:43:20 on dist-test-slave-tm1g"
server_key: "3d22cfdd9cc21b8bfb354c0f330cbabf"
server_key_iv: "ed2bbc4b67a89e9bff9fc347f914f074"
server_key_version: "encryptionkey@0"
I20260806 20:43:20.217839 14578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "f95ba709c0c349b2a92db46aa37b2bea"
format_stamp: "Formatted at 2026-08-06 20:43:20 on dist-test-slave-tm1g"
server_key: "3d22cfdd9cc21b8bfb354c0f330cbabf"
server_key_iv: "ed2bbc4b67a89e9bff9fc347f914f074"
server_key_version: "encryptionkey@0"
I20260806 20:43:20.222941 14578 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260806 20:43:20.226249 14593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:20.227560 14578 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:43:20.227741 14578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "f95ba709c0c349b2a92db46aa37b2bea"
format_stamp: "Formatted at 2026-08-06 20:43:20 on dist-test-slave-tm1g"
server_key: "3d22cfdd9cc21b8bfb354c0f330cbabf"
server_key_iv: "ed2bbc4b67a89e9bff9fc347f914f074"
server_key_version: "encryptionkey@0"
I20260806 20:43:20.227938 14578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:20.243563 14578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:20.247272 14578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:20.247552 14578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:20.265822 14578 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:40573
I20260806 20:43:20.265846 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:40573 every 8 connection(s)
I20260806 20:43:20.267179 14578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:20.270792 14647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:20.271126 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14578
I20260806 20:43:20.271327 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:20.271878 24352 external_mini_cluster.cc:1428] Setting key 1708e5f7b6e831a1d11f662519269095
I20260806 20:43:20.284515 14647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea: Bootstrap starting.
I20260806 20:43:20.284832 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:20.273606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46349 (local address 127.23.200.62:40573)
0806 20:43:20.274459 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:20.274470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:20.274509 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:43:20.275997 (+ 1488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:20.276015 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:20.276387 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:20.276703 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:20.277275 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:20.278631 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:20.280465 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:20.281152 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:20.281285 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:20.282588 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:20.282597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:20.282609 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:20.283135 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:20.283205 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:20.283234 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:20.283346 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:20.283688 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 20:43:20.283867 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":375,"threads_started":1}
I20260806 20:43:20.288146 14647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:20.288616 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:20.289362 14647 log.cc:824] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:20.292601 14647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea: No bootstrap required, opened a new log
I20260806 20:43:20.300545 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:20.293722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36877 (local address 127.23.200.62:40573)
0806 20:43:20.293904 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:20.293910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:20.294449 (+ 539us) server_negotiation.cc:407] Connection header received
0806 20:43:20.294592 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:20.294598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:20.294678 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:20.294783 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:20.295246 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:20.296464 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:20.297853 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:20.298580 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:20.298716 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:20.299766 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:20.299773 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:20.299778 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:20.299805 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:20.299856 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:20.299875 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:20.299988 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:20.300158 (+ 170us) server_negotiation.cc:299] Negotiation successful
0806 20:43:20.300207 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260806 20:43:20.302527 14647 raft_consensus.cc:348] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95ba709c0c349b2a92db46aa37b2bea" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40573 } }
I20260806 20:43:20.302894 14647 raft_consensus.cc:374] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:20.303015 14647 raft_consensus.cc:729] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f95ba709c0c349b2a92db46aa37b2bea, State: Initialized, Role: FOLLOWER
I20260806 20:43:20.303601 14647 consensus_queue.cc:260] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95ba709c0c349b2a92db46aa37b2bea" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40573 } }
I20260806 20:43:20.303838 14647 raft_consensus.cc:388] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:20.304000 14647 raft_consensus.cc:482] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:20.304174 14647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:20.306800 14647 raft_consensus.cc:504] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95ba709c0c349b2a92db46aa37b2bea" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40573 } }
I20260806 20:43:20.307300 14647 leader_election.cc:302] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f95ba709c0c349b2a92db46aa37b2bea; no voters:
I20260806 20:43:20.307875 14647 leader_election.cc:288] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:20.308035 14658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:20.308768 14658 raft_consensus.cc:686] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [term 1 LEADER]: Becoming Leader. State: Replica: f95ba709c0c349b2a92db46aa37b2bea, State: Running, Role: LEADER
I20260806 20:43:20.309383 14658 consensus_queue.cc:237] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95ba709c0c349b2a92db46aa37b2bea" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40573 } }
I20260806 20:43:20.310048 14647 sys_catalog.cc:564] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:20.314191 14659 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f95ba709c0c349b2a92db46aa37b2bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95ba709c0c349b2a92db46aa37b2bea" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40573 } } }
I20260806 20:43:20.314832 14659 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:20.315474 14661 sys_catalog.cc:455] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [sys.catalog]: SysCatalogTable state changed. Reason: New leader f95ba709c0c349b2a92db46aa37b2bea. Latest consensus state: current_term: 1 leader_uuid: "f95ba709c0c349b2a92db46aa37b2bea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f95ba709c0c349b2a92db46aa37b2bea" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40573 } } }
I20260806 20:43:20.315994 14661 sys_catalog.cc:458] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:20.323887 14668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:20.328150 14668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:20.336546 14668 catalog_manager.cc:1347] Generated new cluster ID: cdcd0589c44147bc83183e6d4b6467ca
I20260806 20:43:20.336699 14668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:20.352456 14668 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:20.353749 14668 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:20.360073 14668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f95ba709c0c349b2a92db46aa37b2bea: Generated new TSK 0
I20260806 20:43:20.360904 14668 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:20.367178 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14578
W20260806 20:43:20.368319 14642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46349
W20260806 20:43:20.369048 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14578: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:20.369179 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14578
W20260806 20:43:20.383972 14562 connection.cc:535] client connection to 127.23.200.62:40573 recv error: Network error: failed to read from TLS socket (remote: 127.23.200.62:40573): Connection reset by peer (error 104)
2026-08-06T20:43:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1885 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:20 dist-test-slave-tm1g krb5kdc[14680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:20 dist-test-slave-tm1g krb5kdc[14680](info): set up 2 sockets
Aug 06 20:43:20 dist-test-slave-tm1g krb5kdc[14680](info): commencing operation
krb5kdc: starting...
W20260806 20:43:21.498528 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:21 dist-test-slave-tm1g krb5kdc[14680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:21.754231 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40757
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:39301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:40757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:22.089232 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:22.089522 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:22.089617 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:22.096747 14693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:22.096856 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:22.096903 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:22.096964 14693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:22.097008 14693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:22.106832 14693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:39301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:40757
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:22.108093 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:22.109467 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:22.121208 14698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:22.121162 14702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:22.121256 14699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:22.123749 14693 server_base.cc:1034] running on GCE node
I20260806 20:43:22.124560 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:22.125957 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:22.127158 14693 hybrid_clock.cc:648] HybridClock initialized: now 1786049002127113 us; error 31 us; skew 500 ppm
I20260806 20:43:22.127514 14693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:22 dist-test-slave-tm1g krb5kdc[14680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049002, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:22.134403 14693 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:22.135880 14693 webserver.cc:460] Webserver started at http://127.23.200.62:36625/ using document root <none> and password file <none>
I20260806 20:43:22.136617 14693 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:22.136766 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:22.137091 14693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:22.140121 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "ccf52a5c3ff443f2890d66b6fb19b523"
format_stamp: "Formatted at 2026-08-06 20:43:22 on dist-test-slave-tm1g"
server_key: "40bd8a129cc18b217d4c627dabc0e6ea"
server_key_iv: "3942fb5104730aa8e4c411141cfa747a"
server_key_version: "encryptionkey@0"
I20260806 20:43:22.140916 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "ccf52a5c3ff443f2890d66b6fb19b523"
format_stamp: "Formatted at 2026-08-06 20:43:22 on dist-test-slave-tm1g"
server_key: "40bd8a129cc18b217d4c627dabc0e6ea"
server_key_iv: "3942fb5104730aa8e4c411141cfa747a"
server_key_version: "encryptionkey@0"
I20260806 20:43:22.145988 14693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:43:22.149403 14709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:22.150713 14693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260806 20:43:22.150889 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "ccf52a5c3ff443f2890d66b6fb19b523"
format_stamp: "Formatted at 2026-08-06 20:43:22 on dist-test-slave-tm1g"
server_key: "40bd8a129cc18b217d4c627dabc0e6ea"
server_key_iv: "3942fb5104730aa8e4c411141cfa747a"
server_key_version: "encryptionkey@0"
I20260806 20:43:22.151095 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:22.156657 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:22.160956 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:22.161252 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:22.179826 14693 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:40757
I20260806 20:43:22.179849 14763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:40757 every 8 connection(s)
I20260806 20:43:22.181269 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:22.184862 14764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:22.189956 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14693
I20260806 20:43:22.190135 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:22.190670 24352 external_mini_cluster.cc:1428] Setting key 6a97a038b6eba10b5766485781eaccc0
I20260806 20:43:22.196282 14764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523: Bootstrap starting.
I20260806 20:43:22.198676 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:22.192503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37165 (local address 127.23.200.62:40757)
0806 20:43:22.193309 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:22.193320 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:22.193363 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:43:22.195147 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:22.195169 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:22.195522 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:22.195898 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:22.196980 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:22.196986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:22.196998 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:22.197013 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:22.197065 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:22.197084 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:22.197250 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:22.197374 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 20:43:22.197586 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":396,"threads_started":1}
I20260806 20:43:22.199879 14764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:22.200920 14764 log.cc:824] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:22.202284 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:22.203792 14764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523: No bootstrap required, opened a new log
I20260806 20:43:22.209187 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:22.206823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46649 (local address 127.23.200.62:40757)
0806 20:43:22.207022 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:22.207027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:22.207324 (+ 297us) server_negotiation.cc:407] Connection header received
0806 20:43:22.207482 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:22.207485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:22.207529 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:22.207606 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:22.208520 (+ 914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:22.208524 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:22.208527 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:22.208545 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:22.208584 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:22.208599 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:22.208689 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:22.208825 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 20:43:22.208871 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 20:43:22.212165 14764 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40757 } }
I20260806 20:43:22.212550 14764 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:22.212656 14764 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccf52a5c3ff443f2890d66b6fb19b523, State: Initialized, Role: FOLLOWER
I20260806 20:43:22.213203 14764 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40757 } }
I20260806 20:43:22.213435 14764 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:22.213599 14764 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:22.213764 14764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:22.215989 14764 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40757 } }
I20260806 20:43:22.216473 14764 leader_election.cc:302] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccf52a5c3ff443f2890d66b6fb19b523; no voters:
I20260806 20:43:22.217065 14764 leader_election.cc:288] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:22.217201 14774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:22.217978 14774 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [term 1 LEADER]: Becoming Leader. State: Replica: ccf52a5c3ff443f2890d66b6fb19b523, State: Running, Role: LEADER
I20260806 20:43:22.218542 14774 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40757 } }
I20260806 20:43:22.219188 14764 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:22.221658 14776 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccf52a5c3ff443f2890d66b6fb19b523. Latest consensus state: current_term: 1 leader_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40757 } } }
I20260806 20:43:22.222136 14776 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:22.221855 14775 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccf52a5c3ff443f2890d66b6fb19b523" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40757 } } }
I20260806 20:43:22.222273 14775 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:22.226894 14779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:22.231949 14779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:22.241690 14779 catalog_manager.cc:1347] Generated new cluster ID: 357b8688f78a4fc4b98adb88ef7ed6d3
I20260806 20:43:22.241838 14779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:22.263180 14779 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:22.263650 14779 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:22.272850 14779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccf52a5c3ff443f2890d66b6fb19b523: Generated new TSK 0
I20260806 20:43:22.273599 14779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:22.287693 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14693
W20260806 20:43:22.289139 14759 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37165
W20260806 20:43:22.289966 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14693: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:22.290130 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14693
2026-08-06T20:43:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1925 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:22 dist-test-slave-tm1g krb5kdc[14795](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:22 dist-test-slave-tm1g krb5kdc[14795](info): set up 2 sockets
Aug 06 20:43:22 dist-test-slave-tm1g krb5kdc[14795](info): commencing operation
krb5kdc: starting...
W20260806 20:43:23.417871 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:23 dist-test-slave-tm1g krb5kdc[14795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:23.663619 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:37885
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:36793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:23.992364 14808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:23.992631 14808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:23.992810 14808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:23.999846 14808 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:23.999972 14808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:24.000072 14808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:24.000133 14808 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:24.000170 14808 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:24.009758 14808 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:36793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:37885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:24.011081 14808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:24.012396 14808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:24.024013 14815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:24.024216 14814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:24.024252 14817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:24.025260 14808 server_base.cc:1034] running on GCE node
I20260806 20:43:24.026013 14808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:24.027469 14808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:24.028674 14808 hybrid_clock.cc:648] HybridClock initialized: now 1786049004028637 us; error 26 us; skew 500 ppm
I20260806 20:43:24.029026 14808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:24 dist-test-slave-tm1g krb5kdc[14795](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049004, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:24.032123 14808 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:24.033591 14808 webserver.cc:460] Webserver started at http://127.23.200.62:36327/ using document root <none> and password file <none>
I20260806 20:43:24.034272 14808 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:24.034484 14808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:24.034812 14808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:24.037684 14808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "415c7a4705854a4da8e9fe1d4143e03c"
format_stamp: "Formatted at 2026-08-06 20:43:24 on dist-test-slave-tm1g"
server_key: "55ee726c2fe8ccb8ccbc52514be6173a"
server_key_iv: "8cb6b27fa8b7017531857ecd637c98b1"
server_key_version: "encryptionkey@0"
I20260806 20:43:24.038489 14808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "415c7a4705854a4da8e9fe1d4143e03c"
format_stamp: "Formatted at 2026-08-06 20:43:24 on dist-test-slave-tm1g"
server_key: "55ee726c2fe8ccb8ccbc52514be6173a"
server_key_iv: "8cb6b27fa8b7017531857ecd637c98b1"
server_key_version: "encryptionkey@0"
I20260806 20:43:24.043704 14808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 20:43:24.047032 14824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:24.048246 14808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:43:24.048442 14808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "415c7a4705854a4da8e9fe1d4143e03c"
format_stamp: "Formatted at 2026-08-06 20:43:24 on dist-test-slave-tm1g"
server_key: "55ee726c2fe8ccb8ccbc52514be6173a"
server_key_iv: "8cb6b27fa8b7017531857ecd637c98b1"
server_key_version: "encryptionkey@0"
I20260806 20:43:24.048641 14808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:24.068517 14808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:24.072243 14808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:24.072530 14808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:24.091099 14808 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:37885
I20260806 20:43:24.091118 14876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:37885 every 8 connection(s)
I20260806 20:43:24.092394 14808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:24.095722 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14808
I20260806 20:43:24.095923 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:24.096342 14877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:24.096529 24352 external_mini_cluster.cc:1428] Setting key 7fc4584605c2e692e696787b61cc3d10
I20260806 20:43:24.110889 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c: Bootstrap starting.
Aug 06 20:43:24 dist-test-slave-tm1g krb5kdc[14795](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:24.113921 14877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:24.115016 14877 log.cc:824] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:24.117708 14877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c: No bootstrap required, opened a new log
I20260806 20:43:24.119360 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:24.098421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35705 (local address 127.23.200.62:37885)
0806 20:43:24.099267 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:24.099282 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:24.099325 (+ 43us) server_negotiation.cc:407] Connection header received
0806 20:43:24.101129 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:24.101150 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:24.101520 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:24.101876 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:24.103655 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:24.104908 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:24.106700 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:24.107367 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:24.107513 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:24.112319 (+ 4806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:24.112326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:24.112338 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:24.112371 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:24.115454 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:24.116119 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:24.116125 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:24.116131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:24.116295 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:24.116700 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:24.116704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:24.116707 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:24.117186 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:24.117511 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:24.117952 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 20:43:24.118444 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":375,"threads_started":1}
I20260806 20:43:24.122960 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:24.125576 14877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415c7a4705854a4da8e9fe1d4143e03c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 37885 } }
I20260806 20:43:24.126070 14877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:24.126190 14877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 415c7a4705854a4da8e9fe1d4143e03c, State: Initialized, Role: FOLLOWER
I20260806 20:43:24.126745 14877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415c7a4705854a4da8e9fe1d4143e03c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 37885 } }
I20260806 20:43:24.126998 14877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:24.127166 14877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:24.127379 14877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:24.129951 14877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415c7a4705854a4da8e9fe1d4143e03c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 37885 } }
I20260806 20:43:24.130550 14877 leader_election.cc:302] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 415c7a4705854a4da8e9fe1d4143e03c; no voters:
I20260806 20:43:24.131168 14877 leader_election.cc:288] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:24.131304 14887 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:24.131953 14887 raft_consensus.cc:686] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [term 1 LEADER]: Becoming Leader. State: Replica: 415c7a4705854a4da8e9fe1d4143e03c, State: Running, Role: LEADER
I20260806 20:43:24.132572 14887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415c7a4705854a4da8e9fe1d4143e03c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 37885 } }
I20260806 20:43:24.135388 14889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "415c7a4705854a4da8e9fe1d4143e03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415c7a4705854a4da8e9fe1d4143e03c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 37885 } } }
I20260806 20:43:24.135797 14889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:24.138001 14889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 415c7a4705854a4da8e9fe1d4143e03c. Latest consensus state: current_term: 1 leader_uuid: "415c7a4705854a4da8e9fe1d4143e03c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "415c7a4705854a4da8e9fe1d4143e03c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 37885 } } }
I20260806 20:43:24.138262 14889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:24.143978 14877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:24.144343 14891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:24.144649 14880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:24.127282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51399 (local address 127.23.200.62:37885)
0806 20:43:24.127475 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:24.127482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:24.127874 (+ 392us) server_negotiation.cc:407] Connection header received
0806 20:43:24.128000 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:24.128005 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:24.128110 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:24.128215 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:24.129551 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:24.130709 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:24.132586 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:24.133233 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:24.133333 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:24.136813 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:24.136823 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:24.136830 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:24.136878 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:24.141312 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:24.142319 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:24.142327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:24.142332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:24.142620 (+ 288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:24.143128 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:24.143136 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:24.143139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:24.143439 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:24.143706 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:24.144036 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 20:43:24.144250 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 20:43:24.150606 14891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:24.160882 14891 catalog_manager.cc:1347] Generated new cluster ID: 1f8ec45aafe94f9788ef6172c0c7b0b8
I20260806 20:43:24.160990 14891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:24.172147 14891 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:24.173550 14891 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:24.182053 14891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 415c7a4705854a4da8e9fe1d4143e03c: Generated new TSK 0
I20260806 20:43:24.182993 14891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:24.196772 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14808
I20260806 20:43:24.197836 14872 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:24.286809 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14808
2026-08-06T20:43:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1995 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:24 dist-test-slave-tm1g krb5kdc[14910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:24 dist-test-slave-tm1g krb5kdc[14910](info): set up 2 sockets
Aug 06 20:43:24 dist-test-slave-tm1g krb5kdc[14910](info): commencing operation
krb5kdc: starting...
W20260806 20:43:25.397972 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:25 dist-test-slave-tm1g krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:25.641558 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46829
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:40033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:46829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:25.974113 14923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:25.974429 14923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:25.974603 14923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:25.981609 14923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:25.981736 14923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:25.981848 14923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:25.981951 14923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:25.982039 14923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:25.991487 14923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:40033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:46829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:46829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:25.992748 14923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:25.994030 14923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:26.005625 14928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:26.005590 14929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:26.006073 14931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:26.006637 14923 server_base.cc:1034] running on GCE node
I20260806 20:43:26.007351 14923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:26.008711 14923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:26.009933 14923 hybrid_clock.cc:648] HybridClock initialized: now 1786049006009904 us; error 41 us; skew 500 ppm
I20260806 20:43:26.010262 14923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:26 dist-test-slave-tm1g krb5kdc[14910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049006, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:26.013269 14923 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:26.014686 14923 webserver.cc:460] Webserver started at http://127.23.200.62:43307/ using document root <none> and password file <none>
I20260806 20:43:26.015345 14923 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:26.015498 14923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:26.015806 14923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:26.018740 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "04886142071d45ae871636463eb5cb3c"
format_stamp: "Formatted at 2026-08-06 20:43:26 on dist-test-slave-tm1g"
server_key: "f456744e887fd7496653cd694c156b3f"
server_key_iv: "c22056d3cea3c2a404b60cd425e4d667"
server_key_version: "encryptionkey@0"
I20260806 20:43:26.019507 14923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "04886142071d45ae871636463eb5cb3c"
format_stamp: "Formatted at 2026-08-06 20:43:26 on dist-test-slave-tm1g"
server_key: "f456744e887fd7496653cd694c156b3f"
server_key_iv: "c22056d3cea3c2a404b60cd425e4d667"
server_key_version: "encryptionkey@0"
I20260806 20:43:26.024317 14923 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260806 20:43:26.027585 14938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:26.028826 14923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:43:26.029007 14923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "04886142071d45ae871636463eb5cb3c"
format_stamp: "Formatted at 2026-08-06 20:43:26 on dist-test-slave-tm1g"
server_key: "f456744e887fd7496653cd694c156b3f"
server_key_iv: "c22056d3cea3c2a404b60cd425e4d667"
server_key_version: "encryptionkey@0"
I20260806 20:43:26.029198 14923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:26.046840 14923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:26.050393 14923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:26.050657 14923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:26.068718 14923 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:46829
I20260806 20:43:26.068758 14990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:46829 every 8 connection(s)
I20260806 20:43:26.070115 14923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:26.073736 14991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:26.073900 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 14923
I20260806 20:43:26.074085 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:26.074672 24352 external_mini_cluster.cc:1428] Setting key de7c5e64a255fd634c79e743663f4115
I20260806 20:43:26.086077 14991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c: Bootstrap starting.
I20260806 20:43:26.087077 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:26.076398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39227 (local address 127.23.200.62:46829)
0806 20:43:26.077143 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:26.077154 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:26.077193 (+ 39us) server_negotiation.cc:407] Connection header received
0806 20:43:26.078846 (+ 1653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:26.078868 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:26.079219 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:26.079588 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:26.080134 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:26.081375 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:26.082760 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:26.083466 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:26.083714 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:26.084898 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:26.084905 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:26.084917 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:26.085421 (+ 504us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:26.085490 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:26.085514 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:26.085640 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:26.085857 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 20:43:26.086106 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":317,"threads_started":1}
I20260806 20:43:26.089223 14991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:26.090402 14991 log.cc:824] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:26.090533 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:26.092938 14991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c: No bootstrap required, opened a new log
I20260806 20:43:26.102687 14991 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04886142071d45ae871636463eb5cb3c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46829 } }
I20260806 20:43:26.103096 14991 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:26.103242 14991 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04886142071d45ae871636463eb5cb3c, State: Initialized, Role: FOLLOWER
I20260806 20:43:26.103852 14991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04886142071d45ae871636463eb5cb3c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46829 } }
I20260806 20:43:26.104153 14991 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:26.104343 14994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:26.094774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53901 (local address 127.23.200.62:46829)
0806 20:43:26.094951 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:26.094955 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:26.098793 (+ 3838us) server_negotiation.cc:407] Connection header received
0806 20:43:26.098982 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:26.098987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:26.099058 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:26.099174 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:26.099698 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:26.100432 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:26.101760 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:26.102206 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:26.102367 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:26.103571 (+ 1204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:26.103576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:26.103579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:26.103594 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:26.103632 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:26.103647 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:26.103741 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:26.103873 (+ 132us) server_negotiation.cc:299] Negotiation successful
0806 20:43:26.103961 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 20:43:26.104351 14991 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:26.104614 14991 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:26.107152 14991 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04886142071d45ae871636463eb5cb3c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46829 } }
I20260806 20:43:26.107662 14991 leader_election.cc:302] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04886142071d45ae871636463eb5cb3c; no voters:
I20260806 20:43:26.108237 14991 leader_election.cc:288] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:26.108381 15001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:26.109081 15001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [term 1 LEADER]: Becoming Leader. State: Replica: 04886142071d45ae871636463eb5cb3c, State: Running, Role: LEADER
I20260806 20:43:26.109597 15001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04886142071d45ae871636463eb5cb3c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46829 } }
I20260806 20:43:26.110349 14991 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:26.112566 15003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04886142071d45ae871636463eb5cb3c. Latest consensus state: current_term: 1 leader_uuid: "04886142071d45ae871636463eb5cb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04886142071d45ae871636463eb5cb3c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46829 } } }
I20260806 20:43:26.113037 15003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:26.112874 15002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04886142071d45ae871636463eb5cb3c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04886142071d45ae871636463eb5cb3c" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 46829 } } }
I20260806 20:43:26.113174 15002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:26.114495 15010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:26.119401 15010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:26.129024 15010 catalog_manager.cc:1347] Generated new cluster ID: 0343df82d8074c2792c10ac7faaa9041
I20260806 20:43:26.129179 15010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:26.154177 15010 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:26.155478 15010 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:26.163586 15010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04886142071d45ae871636463eb5cb3c: Generated new TSK 0
I20260806 20:43:26.164326 15010 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:26.178020 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14923
W20260806 20:43:26.179500 14986 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:39227
W20260806 20:43:26.180150 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 14923: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:26.180292 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 14923
2026-08-06T20:43:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1893 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:26 dist-test-slave-tm1g krb5kdc[15023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:26 dist-test-slave-tm1g krb5kdc[15023](info): set up 2 sockets
Aug 06 20:43:26 dist-test-slave-tm1g krb5kdc[15023](info): commencing operation
krb5kdc: starting...
W20260806 20:43:27.310261 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:27 dist-test-slave-tm1g krb5kdc[15023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049007, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:27.556838 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40905
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:33593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:40905
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:27.885110 15036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:27.885411 15036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:27.885581 15036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:27.892620 15036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:27.892717 15036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:27.892831 15036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:27.892946 15036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:27.893025 15036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:27.902561 15036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:33593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:40905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:27.903859 15036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:27.905169 15036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:27.917059 15041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:27.917120 15044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:27.917030 15042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:27.917870 15036 server_base.cc:1034] running on GCE node
I20260806 20:43:27.918664 15036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:27.919922 15036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:27.921126 15036 hybrid_clock.cc:648] HybridClock initialized: now 1786049007921087 us; error 29 us; skew 500 ppm
I20260806 20:43:27.921463 15036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:27 dist-test-slave-tm1g krb5kdc[15023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:27.931911 15036 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:27.933369 15036 webserver.cc:460] Webserver started at http://127.23.200.62:33781/ using document root <none> and password file <none>
I20260806 20:43:27.934052 15036 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:27.934199 15036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:27.934577 15036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:27.937543 15036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "7152da46f83f4faba9e8124d9d2959a5"
format_stamp: "Formatted at 2026-08-06 20:43:27 on dist-test-slave-tm1g"
server_key: "95c68d61498304c49636fdb066758b17"
server_key_iv: "c3c385ac2101bfbf6d9a28af7bec12be"
server_key_version: "encryptionkey@0"
I20260806 20:43:27.938338 15036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "7152da46f83f4faba9e8124d9d2959a5"
format_stamp: "Formatted at 2026-08-06 20:43:27 on dist-test-slave-tm1g"
server_key: "95c68d61498304c49636fdb066758b17"
server_key_iv: "c3c385ac2101bfbf6d9a28af7bec12be"
server_key_version: "encryptionkey@0"
I20260806 20:43:27.943495 15036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:43:27.946820 15051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:27.948052 15036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 20:43:27.948239 15036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "7152da46f83f4faba9e8124d9d2959a5"
format_stamp: "Formatted at 2026-08-06 20:43:27 on dist-test-slave-tm1g"
server_key: "95c68d61498304c49636fdb066758b17"
server_key_iv: "c3c385ac2101bfbf6d9a28af7bec12be"
server_key_version: "encryptionkey@0"
I20260806 20:43:27.948436 15036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:27.954066 15036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:27.957813 15036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:27.958101 15036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:27.976481 15036 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:40905
I20260806 20:43:27.976511 15103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:40905 every 8 connection(s)
I20260806 20:43:27.977857 15036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:27.979688 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15036
I20260806 20:43:27.979874 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:27.980466 24352 external_mini_cluster.cc:1428] Setting key bfeca74b63a92eeebc1cd79a4c5fa13d
I20260806 20:43:27.982445 15104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:27.989183 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:27.982213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34117 (local address 127.23.200.62:40905)
0806 20:43:27.983087 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:27.983103 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:27.983150 (+ 47us) server_negotiation.cc:407] Connection header received
0806 20:43:27.985344 (+ 2194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:27.985369 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:27.985807 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:27.986246 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:27.987249 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:27.987256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:27.987272 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:27.987300 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:27.987393 (+ 93us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:27.987422 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:27.987637 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:27.987782 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 20:43:27.987975 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":351,"threads_started":1}
I20260806 20:43:27.992913 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:27.996099 15104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5: Bootstrap starting.
I20260806 20:43:27.999780 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:27.997250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42555 (local address 127.23.200.62:40905)
0806 20:43:27.997476 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:27.997481 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:27.997879 (+ 398us) server_negotiation.cc:407] Connection header received
0806 20:43:27.998056 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:27.998060 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:27.998105 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:27.998180 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:27.999132 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:27.999136 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:27.999139 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 20:43:27.999154 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:27.999192 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 20:43:27.999206 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:27.999292 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:27.999433 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 20:43:27.999483 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 20:43:27.999992 15104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:28.001080 15104 log.cc:824] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:28.003669 15104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5: No bootstrap required, opened a new log
I20260806 20:43:28.010798 15104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7152da46f83f4faba9e8124d9d2959a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40905 } }
I20260806 20:43:28.011193 15104 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 20:43:28.011312 15104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7152da46f83f4faba9e8124d9d2959a5, State: Initialized, Role: FOLLOWER
I20260806 20:43:28.011847 15104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7152da46f83f4faba9e8124d9d2959a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40905 } }
I20260806 20:43:28.012087 15104 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 20:43:28.012236 15104 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 20:43:28.012404 15104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:28.014555 15104 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7152da46f83f4faba9e8124d9d2959a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40905 } }
I20260806 20:43:28.015030 15104 leader_election.cc:302] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7152da46f83f4faba9e8124d9d2959a5; no voters:
I20260806 20:43:28.015676 15104 leader_election.cc:288] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 20:43:28.015815 15114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:28.016574 15114 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [term 1 LEADER]: Becoming Leader. State: Replica: 7152da46f83f4faba9e8124d9d2959a5, State: Running, Role: LEADER
I20260806 20:43:28.017210 15114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7152da46f83f4faba9e8124d9d2959a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40905 } }
I20260806 20:43:28.017865 15104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:28.020560 15115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7152da46f83f4faba9e8124d9d2959a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7152da46f83f4faba9e8124d9d2959a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40905 } } }
I20260806 20:43:28.020534 15116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7152da46f83f4faba9e8124d9d2959a5. Latest consensus state: current_term: 1 leader_uuid: "7152da46f83f4faba9e8124d9d2959a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7152da46f83f4faba9e8124d9d2959a5" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40905 } } }
I20260806 20:43:28.021024 15115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:28.021072 15116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5 [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:28.022747 15122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:28.028995 15122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:28.037807 15122 catalog_manager.cc:1347] Generated new cluster ID: d5f31eadfd46496b92b3197f8e006dbe
I20260806 20:43:28.037914 15122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:28.059999 15122 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:28.060513 15122 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:28.069773 15122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7152da46f83f4faba9e8124d9d2959a5: Generated new TSK 0
I20260806 20:43:28.070745 15122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:28.076342 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15036
W20260806 20:43:28.077495 15099 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34117
W20260806 20:43:28.078241 24352 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15036: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 20:43:28.078441 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15036
2026-08-06T20:43:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1897 ms)
[----------] 18 tests from AuthTokenIssuingTest (35007 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:28 dist-test-slave-tm1g krb5kdc[15135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:28 dist-test-slave-tm1g krb5kdc[15135](info): set up 2 sockets
Aug 06 20:43:28 dist-test-slave-tm1g krb5kdc[15135](info): commencing operation
krb5kdc: starting...
W20260806 20:43:29.202677 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:29 dist-test-slave-tm1g krb5kdc[15135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:29.457799 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40651
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:40651,127.23.200.61:45597,127.23.200.60:39059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:29.783667 15148 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 20:43:29.783998 15148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:29.784126 15148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:29.784263 15148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:29.791363 15148 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:29.791451 15148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:29.791559 15148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:29.791636 15148 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:29.791749 15148 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:29.801419 15148 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:40651,127.23.200.61:45597,127.23.200.60:39059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:40651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:29.802803 15148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:29.804111 15148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:29.816084 15154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:29.816162 15157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:29.816130 15155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:29.817211 15148 server_base.cc:1034] running on GCE node
I20260806 20:43:29.817889 15148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:29.819235 15148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:29.820454 15148 hybrid_clock.cc:648] HybridClock initialized: now 1786049009820392 us; error 32 us; skew 500 ppm
I20260806 20:43:29.820786 15148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:29 dist-test-slave-tm1g krb5kdc[15135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:29.823877 15148 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:29.825322 15148 webserver.cc:460] Webserver started at http://127.23.200.62:36507/ using document root <none> and password file <none>
I20260806 20:43:29.826004 15148 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:29.826135 15148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:29.826526 15148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:29.829393 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "7416b9eef51b4cca8b6097ae46eb5d3b"
format_stamp: "Formatted at 2026-08-06 20:43:29 on dist-test-slave-tm1g"
server_key: "78dda524c7ec65aa9498d03249b1b62d"
server_key_iv: "75cad1c4c7f5441a1f7f1f0f8f90e59e"
server_key_version: "encryptionkey@0"
I20260806 20:43:29.830183 15148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "7416b9eef51b4cca8b6097ae46eb5d3b"
format_stamp: "Formatted at 2026-08-06 20:43:29 on dist-test-slave-tm1g"
server_key: "78dda524c7ec65aa9498d03249b1b62d"
server_key_iv: "75cad1c4c7f5441a1f7f1f0f8f90e59e"
server_key_version: "encryptionkey@0"
I20260806 20:43:29.835098 15148 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 20:43:29.838455 15164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:29.839684 15148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:43:29.839874 15148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "7416b9eef51b4cca8b6097ae46eb5d3b"
format_stamp: "Formatted at 2026-08-06 20:43:29 on dist-test-slave-tm1g"
server_key: "78dda524c7ec65aa9498d03249b1b62d"
server_key_iv: "75cad1c4c7f5441a1f7f1f0f8f90e59e"
server_key_version: "encryptionkey@0"
I20260806 20:43:29.840083 15148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:29.854635 15148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:29.858507 15148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:29.858875 15148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:29.877107 15148 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:40651
I20260806 20:43:29.877139 15215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:40651 every 8 connection(s)
I20260806 20:43:29.878372 15148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:29.880117 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15148
I20260806 20:43:29.880327 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:29.880975 24352 external_mini_cluster.cc:1428] Setting key 52f78f0eedc64f80beb2fa18639b9c07
I20260806 20:43:29.881824 15216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:29.888053 15216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:29 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.61@KRBTEST.COM" created.
I20260806 20:43:29.909512 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:29.892280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.23.200.62:40651)
0806 20:43:29.892985 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:29.892992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:29.893039 (+ 47us) server_negotiation.cc:407] Connection header received
0806 20:43:29.894121 (+ 1082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:29.894126 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:29.894384 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:29.894575 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:29.895762 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:29.896882 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:29.898230 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:29.898850 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:29.899046 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:29.902190 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:29.902197 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:29.902207 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:29.902233 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:29.905191 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:29.905899 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:29.905909 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:29.905914 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:29.906058 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:29.906504 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:29.906511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:29.906515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:29.907038 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:29.907272 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:29.907815 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 20:43:29.908122 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":338,"threads_started":1}
I20260806 20:43:29.909559 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:29.891079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:40651 (local address 127.0.0.1:60592)
0806 20:43:29.891785 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:29.891821 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:29.893782 (+ 1961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:29.894590 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:29.894596 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:29.895248 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:29.895560 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:29.895575 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:29.897052 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:29.897061 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:29.898095 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:29.898107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:29.899080 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:29.899087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:29.899281 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:29.899941 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:29.899962 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:29.901960 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:29.905322 (+ 3362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:29.905336 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:29.905348 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:29.905781 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:29.906174 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:29.906178 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:29.906182 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:29.906395 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:29.907213 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:29.907221 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:29.907456 (+ 235us) client_negotiation.cc:769] Sending connection context
0806 20:43:29.907696 (+ 240us) client_negotiation.cc:240] Negotiation successful
0806 20:43:29.908261 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"mutex_wait_us":139,"thread_start_us":299,"threads_started":1}
I20260806 20:43:29.912559 15216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:29.913587 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:29.913203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43606)
0806 20:43:29.913365 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:29.913474 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 20:43:29.913818 15168 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
I20260806 20:43:29.915390 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:29.915056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43610)
0806 20:43:29.915204 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:29.915269 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 20:43:29.915673 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:45597: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab.
Entry for principal kudu/127.23.200.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab.
I20260806 20:43:29.937402 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:45597
--webserver_interface=127.23.200.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab
--principal=kudu/127.23.200.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:40651,127.23.200.61:45597,127.23.200.60:39059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:43:29.966809 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } attempt: 1
I20260806 20:43:29.968160 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:29.967592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43622)
0806 20:43:29.967821 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:29.967979 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 20:43:29.969374 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:29.968935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43634)
0806 20:43:29.969126 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:29.969219 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260806 20:43:29.969713 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:45597: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
I20260806 20:43:30.038002 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } attempt: 2
I20260806 20:43:30.039283 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.038768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43644)
0806 20:43:30.039041 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.039132 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 20:43:30.040288 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.039928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43654)
0806 20:43:30.040096 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.040166 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:43:30.040477 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:45597: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
I20260806 20:43:30.131771 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } attempt: 3
I20260806 20:43:30.132970 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.132519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43670)
0806 20:43:30.132731 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.132834 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 20:43:30.134019 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.133653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43686)
0806 20:43:30.133799 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.133873 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 20:43:30.134214 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:45597: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
W20260806 20:43:30.266134 15223 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 20:43:30.266516 15223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:30.266649 15223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:30.266791 15223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:30.273821 15223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:30.273938 15223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:30.274070 15223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:30.274156 15223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:30.274246 15223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:30.277508 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } attempt: 4
I20260806 20:43:30.278702 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.278143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43688)
0806 20:43:30.278474 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.278567 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260806 20:43:30.279713 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.279336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43698)
0806 20:43:30.279496 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.279587 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 20:43:30.279913 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:45597: Network error: Client connection negotiation failed: client connection to 127.23.200.61:45597: connect: Connection refused (error 111)
I20260806 20:43:30.284062 15223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:40651,127.23.200.61:45597,127.23.200.60:39059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab
--principal=kudu/127.23.200.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:45597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.200.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:30.285329 15223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:30.286724 15223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:30.298187 15231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:30.298271 15229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:30.298247 15228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:30.299199 15223 server_base.cc:1034] running on GCE node
I20260806 20:43:30.299923 15223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:30.301294 15223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:30.302546 15223 hybrid_clock.cc:648] HybridClock initialized: now 1786049010302479 us; error 38 us; skew 500 ppm
I20260806 20:43:30.302881 15223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:30.306090 15223 init.cc:377] Logged in from keytab as kudu/127.23.200.61@KRBTEST.COM (short username kudu)
I20260806 20:43:30.307575 15223 webserver.cc:460] Webserver started at http://127.23.200.61:44309/ using document root <none> and password file <none>
I20260806 20:43:30.308280 15223 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:30.308413 15223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:30.308743 15223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:30.311681 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data/instance:
uuid: "1011c68026224265af6886829080a18a"
format_stamp: "Formatted at 2026-08-06 20:43:30 on dist-test-slave-tm1g"
server_key: "e9e5e56c5e1352564ce3b129c02ff4a1"
server_key_iv: "e35c2912c540c069607cb8f5e20bfbd0"
server_key_version: "encryptionkey@0"
I20260806 20:43:30.312498 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal/instance:
uuid: "1011c68026224265af6886829080a18a"
format_stamp: "Formatted at 2026-08-06 20:43:30 on dist-test-slave-tm1g"
server_key: "e9e5e56c5e1352564ce3b129c02ff4a1"
server_key_iv: "e35c2912c540c069607cb8f5e20bfbd0"
server_key_version: "encryptionkey@0"
I20260806 20:43:30.317595 15223 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 20:43:30.320967 15238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:30.322182 15223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 20:43:30.322386 15223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal
uuid: "1011c68026224265af6886829080a18a"
format_stamp: "Formatted at 2026-08-06 20:43:30 on dist-test-slave-tm1g"
server_key: "e9e5e56c5e1352564ce3b129c02ff4a1"
server_key_iv: "e35c2912c540c069607cb8f5e20bfbd0"
server_key_version: "encryptionkey@0"
I20260806 20:43:30.322613 15223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:30.347865 15223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:30.352954 15223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:30.353240 15223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:30.371374 15223 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.61:45597
I20260806 20:43:30.371393 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.61:45597 every 8 connection(s)
I20260806 20:43:30.372669 15223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
I20260806 20:43:30.373091 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15223
I20260806 20:43:30.373382 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-1/wal/instance
I20260806 20:43:30.374044 24352 external_mini_cluster.cc:1428] Setting key c3cfcf467439787c66c99b03ea05de8b
I20260806 20:43:30.376497 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:30.382764 15292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.200.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.60@KRBTEST.COM" created.
I20260806 20:43:30.403642 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.385330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60596 (local address 127.23.200.62:40651)
0806 20:43:30.385514 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.385518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.386805 (+ 1287us) server_negotiation.cc:407] Connection header received
0806 20:43:30.388626 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.388632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.388725 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.388837 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.390265 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.391006 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.392488 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.392909 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.393043 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.397107 (+ 4064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.397113 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.397117 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.397160 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.400171 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.401105 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.401111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.401115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.401274 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.401657 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.401665 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.401672 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.402093 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.402258 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.402926 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.403194 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:43:30.404193 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.385862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:40651 (local address 127.0.0.1:60596)
0806 20:43:30.386641 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.386679 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:30.388349 (+ 1670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:30.389069 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:30.389076 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:30.389568 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:30.390087 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.390103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.391174 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.391178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.392326 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.392339 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.393131 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.393138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.393373 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.394054 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:30.394077 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:30.396823 (+ 2746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:30.400328 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.400345 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.400356 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.400990 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.401383 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.401386 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.401389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.401547 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.402220 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:30.402225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:30.402575 (+ 350us) client_negotiation.cc:769] Sending connection context
0806 20:43:30.402796 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 20:43:30.403154 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":288,"threads_started":1}
I20260806 20:43:30.405911 15292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:43:30.421756 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.406877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43710 (local address 127.23.200.61:45597)
0806 20:43:30.407507 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.407513 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.407537 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:43:30.407576 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.407581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.407733 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.407842 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.408768 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.410149 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.411500 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.412463 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.412599 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.415937 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.415949 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.415961 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.415988 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.418784 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.419624 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.419631 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.419633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.419749 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.420187 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.420190 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.420192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.420524 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.420687 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.421132 (+ 445us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.421465 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":131,"server-negotiator.queue_time_us":494,"thread_start_us":356,"threads_started":1}
I20260806 20:43:30.421835 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.406573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43710)
0806 20:43:30.406732 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.406754 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:30.406830 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:30.407899 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:30.407907 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:30.408392 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:30.408594 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.408608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.410298 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.410345 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.411348 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.411367 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.412606 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.412610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.412741 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.413278 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:30.413295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:30.415728 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:30.418922 (+ 3194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.418930 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.418936 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.419480 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.419866 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.419874 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.419880 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.420071 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.420663 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:30.420670 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:30.420840 (+ 170us) client_negotiation.cc:769] Sending connection context
0806 20:43:30.421014 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 20:43:30.421347 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260806 20:43:30.424315 15292 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:30.425524 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.425066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51208)
0806 20:43:30.425217 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.425362 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 20:43:30.425799 15239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.427450 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.427077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51212)
0806 20:43:30.427224 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.427298 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:43:30.427665 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab.
Entry for principal kudu/127.23.200.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab.
I20260806 20:43:30.433655 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:39059
--webserver_interface=127.23.200.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab
--principal=kudu/127.23.200.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:43789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:40651,127.23.200.61:45597,127.23.200.60:39059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:43:30.477015 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 1
I20260806 20:43:30.478220 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.477760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51220)
0806 20:43:30.477961 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.478072 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 20:43:30.479207 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.478861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51234)
0806 20:43:30.479030 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.479101 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 20:43:30.479411 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.547607 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 2
I20260806 20:43:30.548687 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.548235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51248)
0806 20:43:30.548432 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.548535 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 20:43:30.549619 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.549296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51264)
0806 20:43:30.549436 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.549509 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 20:43:30.549825 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.579180 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } attempt: 5
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:43:30.594439 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.580051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.23.200.61:45597)
0806 20:43:30.580217 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.580221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.580282 (+ 61us) server_negotiation.cc:407] Connection header received
0806 20:43:30.580480 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.580484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.580563 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.580692 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.581860 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.582627 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.584316 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.584750 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.584842 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.588856 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.588861 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.588865 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.588896 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.591596 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.592500 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.592504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.592506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.592622 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.593068 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.593071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.593073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.593337 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.593446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.593961 (+ 515us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.594126 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:43:30.594759 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.579929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43724)
0806 20:43:30.580167 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.580195 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:30.580352 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:30.580773 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:30.580779 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:30.581371 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:30.581685 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.581702 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.582765 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.582770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.584174 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.584189 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.584899 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.584905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.585117 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.585795 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:30.585818 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:30.588660 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:30.591730 (+ 3070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.591735 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.591739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.592371 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.592741 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.592748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.592753 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.592954 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.593477 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:30.593483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:30.593691 (+ 208us) client_negotiation.cc:769] Sending connection context
0806 20:43:30.593877 (+ 186us) client_negotiation.cc:240] Negotiation successful
0806 20:43:30.594229 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260806 20:43:30.595821 15216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:30.596994 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.596641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51274)
0806 20:43:30.596797 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.596869 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260806 20:43:30.598114 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.597808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51280)
0806 20:43:30.597941 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.598022 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:43:30.598395 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.641042 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 3
I20260806 20:43:30.642194 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.641703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51288)
0806 20:43:30.641923 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.642024 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 20:43:30.643292 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.642858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51302)
0806 20:43:30.643092 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.643176 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260806 20:43:30.643527 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.649626 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 1
I20260806 20:43:30.650648 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.650217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51310)
0806 20:43:30.650426 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.650505 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 20:43:30.651594 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.651270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51324)
0806 20:43:30.651419 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.651494 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 20:43:30.651791 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.720006 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 2
I20260806 20:43:30.721187 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.720685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51336)
0806 20:43:30.720934 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.721024 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 20:43:30.722278 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.721912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51352)
0806 20:43:30.722071 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.722161 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 20:43:30.722527 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
W20260806 20:43:30.767385 15299 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 20:43:30.767771 15299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:30.768018 15299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:30.768247 15299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:30.775985 15299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:30.776114 15299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:30.776263 15299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:30.776363 15299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:30.776476 15299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:30.785955 15299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:43789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:40651,127.23.200.61:45597,127.23.200.60:39059
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab
--principal=kudu/127.23.200.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:39059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.200.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:30.786727 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 4
I20260806 20:43:30.787338 15299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:30.787909 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.787414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51368)
0806 20:43:30.787660 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.787745 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260806 20:43:30.788763 15299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 20:43:30.788913 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.788569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51372)
0806 20:43:30.788710 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.788784 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 20:43:30.789119 15292 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
W20260806 20:43:30.800669 15304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:30.800700 15307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:30.800626 15305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:30.801575 15299 server_base.cc:1034] running on GCE node
I20260806 20:43:30.802373 15299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:30.803746 15299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:30.804998 15299 hybrid_clock.cc:648] HybridClock initialized: now 1786049010804924 us; error 54 us; skew 500 ppm
I20260806 20:43:30.805316 15299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:30.808463 15299 init.cc:377] Logged in from keytab as kudu/127.23.200.60@KRBTEST.COM (short username kudu)
I20260806 20:43:30.809764 15299 webserver.cc:460] Webserver started at http://127.23.200.60:36631/ using document root <none> and password file <none>
I20260806 20:43:30.810438 15299 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:30.810526 15299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:30.810843 15299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:30.813686 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data/instance:
uuid: "06598d7a2e6a415b9efaa6ad86a28492"
format_stamp: "Formatted at 2026-08-06 20:43:30 on dist-test-slave-tm1g"
server_key: "028ec796404b6562e40c6a05fc9b087b"
server_key_iv: "c9ae2addf356bc50234e1ed3fb2edb3a"
server_key_version: "encryptionkey@0"
I20260806 20:43:30.814522 15299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal/instance:
uuid: "06598d7a2e6a415b9efaa6ad86a28492"
format_stamp: "Formatted at 2026-08-06 20:43:30 on dist-test-slave-tm1g"
server_key: "028ec796404b6562e40c6a05fc9b087b"
server_key_iv: "c9ae2addf356bc50234e1ed3fb2edb3a"
server_key_version: "encryptionkey@0"
I20260806 20:43:30.819564 15299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 20:43:30.822795 15314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:30.824120 15299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260806 20:43:30.824260 15299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal
uuid: "06598d7a2e6a415b9efaa6ad86a28492"
format_stamp: "Formatted at 2026-08-06 20:43:30 on dist-test-slave-tm1g"
server_key: "028ec796404b6562e40c6a05fc9b087b"
server_key_iv: "c9ae2addf356bc50234e1ed3fb2edb3a"
server_key_version: "encryptionkey@0"
I20260806 20:43:30.824435 15299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:30.828787 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 3
I20260806 20:43:30.830092 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.829449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51388)
0806 20:43:30.829754 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.829871 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 20:43:30.831238 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.830851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51396)
0806 20:43:30.831026 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.831119 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 20:43:30.831432 15216 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:39059: Network error: Client connection negotiation failed: client connection to 127.23.200.60:39059: connect: Connection refused (error 111)
I20260806 20:43:30.876299 15299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:30.880962 15299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:30.881260 15299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:30.899873 15299 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.60:39059
I20260806 20:43:30.899886 15366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.60:39059 every 8 connection(s)
I20260806 20:43:30.901248 15299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
I20260806 20:43:30.904839 15367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:30.908469 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15299
I20260806 20:43:30.908689 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786048651485794-24352-0/minicluster-data/master-2/wal/instance
I20260806 20:43:30.909142 24352 external_mini_cluster.cc:1428] Setting key 28a4edbc6a614f48ce26402fd6b12251
I20260806 20:43:30.911981 15367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:30.932466 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.915913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60612 (local address 127.23.200.62:40651)
0806 20:43:30.916614 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.916621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.917185 (+ 564us) server_negotiation.cc:407] Connection header received
0806 20:43:30.917355 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.917360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.917433 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.917558 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.918671 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.919621 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.921348 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.922088 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.922231 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.925753 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.925760 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.925767 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.925808 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.929301 (+ 3493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.930179 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.930184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.930188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.930368 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.930745 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.930750 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.930753 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.931100 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.931264 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.931825 (+ 561us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.932043 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":434,"threads_started":1}
I20260806 20:43:30.932991 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.916011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:40651 (local address 127.0.0.1:60612)
0806 20:43:30.917040 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.917084 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:30.917231 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:30.917604 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:30.917608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:30.918193 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:30.918521 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.918532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.919775 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.919780 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.921204 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.921216 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.922290 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.922294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.922484 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.923106 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:30.923136 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:30.925543 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:30.929481 (+ 3938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.929489 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.929496 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.930068 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.930484 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.930487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.930489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.930633 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.931250 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:30.931259 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:30.931456 (+ 197us) client_negotiation.cc:769] Sending connection context
0806 20:43:30.931681 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 20:43:30.931996 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"thread_start_us":619,"threads_started":1}
I20260806 20:43:30.933017 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.910955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51412 (local address 127.23.200.60:39059)
0806 20:43:30.911719 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.911730 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.911770 (+ 40us) server_negotiation.cc:407] Connection header received
0806 20:43:30.913297 (+ 1527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.913317 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.913691 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.914045 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.915799 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.917676 (+ 1877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.919229 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.920015 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.920189 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.924691 (+ 4502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.924699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.924709 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.924742 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.928663 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.929427 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.929437 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.929445 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.929622 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.930019 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.930022 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.930024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.930755 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.931068 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.931428 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.931799 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"spinlock_wait_cycles":32000,"thread_start_us":365,"threads_started":1}
I20260806 20:43:30.934684 15367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:30.948783 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.935606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43728 (local address 127.23.200.61:45597)
0806 20:43:30.935795 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.935800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.935814 (+ 14us) server_negotiation.cc:407] Connection header received
0806 20:43:30.935893 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.935901 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.935989 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.936093 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.936955 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.937637 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.939016 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.939404 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.939495 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.942783 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.942788 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.942791 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.942822 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.946027 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.946725 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.946737 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.946739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.946872 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.947264 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.947268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.947271 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.947756 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.947946 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.948356 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.948525 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:43:30.948897 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.935279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:45597 (local address 127.0.0.1:43728)
0806 20:43:30.935544 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.935563 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:30.935653 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:30.936300 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:30.936305 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:30.936642 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:30.936803 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.936816 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.937780 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.937785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.938856 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.938870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.939550 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.939556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.939761 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.940357 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:30.940373 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:30.942592 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:30.946154 (+ 3562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.946161 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.946165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.946598 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.946989 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.946995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.946998 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.947141 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.947907 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:30.947912 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:30.948079 (+ 167us) client_negotiation.cc:769] Sending connection context
0806 20:43:30.948230 (+ 151us) client_negotiation.cc:240] Negotiation successful
0806 20:43:30.948509 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260806 20:43:30.949961 15367 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:30.951167 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.936174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.23.200.62:40651)
0806 20:43:30.936597 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.936601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.936613 (+ 12us) server_negotiation.cc:407] Connection header received
0806 20:43:30.936648 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.936651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.936710 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.936778 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.938030 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.938717 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.940205 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.940683 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.940778 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.945458 (+ 4680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.945462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.945465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.945494 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.948096 (+ 2602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.948953 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.948959 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.948961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.949106 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.949731 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.949738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.949744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.950050 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.950358 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.950657 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.950883 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333}
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
Aug 06 20:43:30 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:43:30.963032 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.950707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.23.200.60:39059)
0806 20:43:30.950866 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.950871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.951174 (+ 303us) server_negotiation.cc:407] Connection header received
0806 20:43:30.951217 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.951221 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.951290 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.951363 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.952300 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.953353 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.954470 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.954903 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.955000 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.958063 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.958071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.958079 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.958106 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.960567 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.961186 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.961191 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.961195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.961305 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.961732 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.961736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.961738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.962008 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.962207 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.962650 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.962807 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:43:30.963477 15373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.950725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51414)
0806 20:43:30.950941 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:30.950964 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:30.951082 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:30.951497 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:30.951500 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:30.951946 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:30.952118 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.952130 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.953453 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.953456 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.954277 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:30.954287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.955059 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.955065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:30.955277 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.955836 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:30.955852 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:30.957883 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:30.960711 (+ 2828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.960718 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.960722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.961063 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.961419 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:30.961424 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:30.961428 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:30.961592 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:30.962174 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:30.962177 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:30.962394 (+ 217us) client_negotiation.cc:769] Sending connection context
0806 20:43:30.962575 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 20:43:30.963003 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":98}
I20260806 20:43:30.967348 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:30.952902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.23.200.61:45597)
0806 20:43:30.953049 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:30.953053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:30.953067 (+ 14us) server_negotiation.cc:407] Connection header received
0806 20:43:30.953201 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:30.953204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:30.953253 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:30.953324 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:30.954521 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.955331 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.956470 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:30.957083 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:30.957175 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:30.961541 (+ 4366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:30.961547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:30.961551 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:30.961586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:30.964869 (+ 3283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.965598 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.965601 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.965603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.965726 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:30.966133 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:30.966136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:30.966138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:30.966437 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:30.966641 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:30.966947 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 20:43:30.967113 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260806 20:43:30.969545 15367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: Bootstrap starting.
I20260806 20:43:30.971974 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:30.973291 15367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:30.974635 15367 log.cc:824] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:30.977385 15367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: No bootstrap required, opened a new log
I20260806 20:43:30.975504 15191 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60618
I20260806 20:43:30.984351 15367 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:30.984678 15367 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06598d7a2e6a415b9efaa6ad86a28492, State: Initialized, Role: FOLLOWER
I20260806 20:43:30.985128 15367 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:30.986052 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:30.986501 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:30.987054 15367 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 20:43:30.996374 15386 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:43:30.996511 15386 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:43:30.994412 15299 master_runner.cc:186] Error getting master registration for 127.23.200.62:40651: OK, instance_id { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" instance_seqno: 1786049009872082 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 20:43:31.005405 15299 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 20:43:31.008597 15216 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 4
Aug 06 20:43:31 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049009, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
I20260806 20:43:31.025374 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.009193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51430)
0806 20:43:31.009400 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:31.009422 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:31.009558 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:31.009865 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:31.009870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:31.010454 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:31.010692 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:31.010708 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.011742 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.011747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:31.013263 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:31.013281 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.014429 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.014436 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:31.014637 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.015305 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:31.015330 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:31.018411 (+ 3081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:31.022496 (+ 4085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:31.022502 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:31.022509 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:31.023006 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:31.023423 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:31.023429 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:31.023435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:31.023598 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:31.024322 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:31.024327 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:31.024522 (+ 195us) client_negotiation.cc:769] Sending connection context
0806 20:43:31.024665 (+ 143us) client_negotiation.cc:240] Negotiation successful
0806 20:43:31.024940 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260806 20:43:31.025439 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.009375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.23.200.60:39059)
0806 20:43:31.009607 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.009612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.009644 (+ 32us) server_negotiation.cc:407] Connection header received
0806 20:43:31.009689 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.009692 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.009748 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.009822 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.010839 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.011606 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.013435 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.014220 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.014370 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.018623 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:31.018635 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:31.018640 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:31.018679 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:31.022296 (+ 3617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.023133 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.023138 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.023142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.023281 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.023711 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.023716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.023719 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.024188 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:31.024372 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:31.024799 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 20:43:31.025040 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:43:31.030714 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: Bootstrap starting.
I20260806 20:43:31.033862 15216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:31.034997 15216 log.cc:824] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:31.037596 15216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: No bootstrap required, opened a new log
I20260806 20:43:31.042610 15216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:31.042994 15216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7416b9eef51b4cca8b6097ae46eb5d3b, State: Initialized, Role: FOLLOWER
I20260806 20:43:31.043429 15216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:31.044452 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.044833 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:31.045543 15216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:31.045749 15191 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:43:31.045996 15191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:31.049621 15191 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:31.052544 15341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "06598d7a2e6a415b9efaa6ad86a28492"
I20260806 20:43:31.053047 15341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:31.054219 15191 leader_election.cc:288] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [CANDIDATE]: Term 1 election: Requested vote from peers 1011c68026224265af6886829080a18a (127.23.200.61:45597), 06598d7a2e6a415b9efaa6ad86a28492 (127.23.200.60:39059)
I20260806 20:43:31.057777 15341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7416b9eef51b4cca8b6097ae46eb5d3b in term 1.
I20260806 20:43:31.060412 15165 leader_election.cc:302] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06598d7a2e6a415b9efaa6ad86a28492, 7416b9eef51b4cca8b6097ae46eb5d3b; no voters:
W20260806 20:43:31.057821 15401 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:43:31.060801 15401 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:43:31.060930 15389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 20:43:31.057523 15148 master_runner.cc:186] Error getting master registration for 127.23.200.61:45597: OK, instance_id { permanent_uuid: "1011c68026224265af6886829080a18a" instance_seqno: 1786049010366153 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 20:43:31.070335 15148 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 20:43:31.074641 15389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [term 1 LEADER]: Becoming Leader. State: Replica: 7416b9eef51b4cca8b6097ae46eb5d3b, State: Running, Role: LEADER
I20260806 20:43:31.075239 15389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:31.079104 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7416b9eef51b4cca8b6097ae46eb5d3b. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.079559 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:31.088307 15292 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } attempt: 5
I20260806 20:43:31.088451 15407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:31.090920 15407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Aug 06 20:43:31 dist-test-slave-tm1g krb5kdc[15135](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
I20260806 20:43:31.105798 15341 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [term 1 FOLLOWER]: Refusing update from remote peer 7416b9eef51b4cca8b6097ae46eb5d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:43:31.110910 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.088958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:39059 (local address 127.0.0.1:51434)
0806 20:43:31.089266 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:31.089290 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:31.089427 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:31.090037 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:31.090045 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:31.090633 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:31.090850 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:31.090863 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.092510 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.092516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:31.093952 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:31.093968 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.095664 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.095671 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:31.095923 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.096720 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:31.096767 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:31.100762 (+ 3995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:31.107523 (+ 6761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:31.107528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:31.107533 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:31.108090 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:31.108571 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:31.108579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:31.108585 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:31.108912 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:31.109734 (+ 822us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:31.109742 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:31.109950 (+ 208us) client_negotiation.cc:769] Sending connection context
0806 20:43:31.110132 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 20:43:31.110505 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168}
W20260806 20:43:31.111580 15170 tablet.cc:2369] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:43:31.113276 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.089045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51434 (local address 127.23.200.60:39059)
0806 20:43:31.089270 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.089277 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.089366 (+ 89us) server_negotiation.cc:407] Connection header received
0806 20:43:31.089591 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.089598 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.089689 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.089806 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.091118 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.092287 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.094523 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.095290 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.095408 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.101113 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:31.101147 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:31.101153 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:31.101226 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:31.107347 (+ 6121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.108240 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.108246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.108249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.108405 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.109102 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.109107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.109111 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.109501 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:31.109654 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:31.110504 (+ 850us) server_negotiation.cc:299] Negotiation successful
0806 20:43:31.111143 (+ 639us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 20:43:31.119609 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: Bootstrap starting.
I20260806 20:43:31.123714 15292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:31.125074 15292 log.cc:824] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:31.129518 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: No bootstrap required, opened a new log
W20260806 20:43:31.133441 15320 tablet.cc:2369] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:43:31.140621 15292 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:31.141014 15292 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1011c68026224265af6886829080a18a, State: Initialized, Role: FOLLOWER
I20260806 20:43:31.141551 15292 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } }
I20260806 20:43:31.143043 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.143914 15292 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:31.144009 15267 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1011c68026224265af6886829080a18a"
I20260806 20:43:31.144976 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:31.144676 15266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:31.147576 15404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:43:31.149262 15266 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [term 1 FOLLOWER]: Refusing update from remote peer 7416b9eef51b4cca8b6097ae46eb5d3b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:43:31.152483 15267 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7416b9eef51b4cca8b6097ae46eb5d3b in term 1.
I20260806 20:43:31.154733 15404 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:43:31.171986 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7416b9eef51b4cca8b6097ae46eb5d3b. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.172389 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:31.178287 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7416b9eef51b4cca8b6097ae46eb5d3b. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.178617 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:31.180881 15410 mvcc.cc:205] Tried to move back new op lower bound from 7315656749469175808 to 7315656749370208256. Current Snapshot: MvccSnapshot[applied={T|T < 7315656749469175808}]
I20260806 20:43:31.181589 15404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.181919 15404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:31.183511 15389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.183797 15389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:31.184063 15407 catalog_manager.cc:1347] Generated new cluster ID: 6f4e35f97d8b4db8a23e22d602cfdbf2
I20260806 20:43:31.184144 15407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:31.198328 15372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.104971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.23.200.62:40651)
0806 20:43:31.105175 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.105185 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.142755 (+ 37570us) server_negotiation.cc:407] Connection header received
0806 20:43:31.143134 (+ 379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.143144 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.143248 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.143332 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.145770 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.146969 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.162603 (+ 15634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.163499 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.163645 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.185945 (+ 22300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:31.185951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:31.185956 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:31.185994 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:31.189598 (+ 3604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.195703 (+ 6105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.195709 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.195712 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.195864 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.196442 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.196448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.196451 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.196789 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:31.197075 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:31.197689 (+ 614us) server_negotiation.cc:299] Negotiation successful
0806 20:43:31.197920 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 20:43:31.202431 15417 mvcc.cc:205] Tried to move back new op lower bound from 7315656749469175808 to 7315656749370208256. Current Snapshot: MvccSnapshot[applied={T|T < 7315656749469175808}]
I20260806 20:43:31.204394 15371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.105390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.23.200.60:39059)
0806 20:43:31.113914 (+ 8524us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.113925 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.150824 (+ 36899us) server_negotiation.cc:407] Connection header received
0806 20:43:31.151086 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.151092 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.151186 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.151263 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.156284 (+ 5021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.157304 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.176028 (+ 18724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.176712 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.176818 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.190493 (+ 13675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:31.190500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:31.190504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:31.190546 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:31.193773 (+ 3227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.194608 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.194613 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.194617 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.194781 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.198857 (+ 4076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.198863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.198867 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.199221 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:31.199470 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:31.203697 (+ 4227us) server_negotiation.cc:299] Negotiation successful
0806 20:43:31.203979 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8243,"thread_start_us":40670,"threads_started":1}
I20260806 20:43:31.205854 15416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.206247 15416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:31.206763 15375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7416b9eef51b4cca8b6097ae46eb5d3b" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 40651 } } peers { permanent_uuid: "1011c68026224265af6886829080a18a" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 45597 } } peers { permanent_uuid: "06598d7a2e6a415b9efaa6ad86a28492" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 39059 } } }
I20260806 20:43:31.207084 15375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:31.216724 15407 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:31.218744 15407 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:31.219198 15296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:31.153990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43750 (local address 127.23.200.61:45597)
0806 20:43:31.154220 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.154230 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.164941 (+ 10711us) server_negotiation.cc:407] Connection header received
0806 20:43:31.173381 (+ 8440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.173387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.173467 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.173539 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.182872 (+ 9333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.183983 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.199536 (+ 15553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:31.200259 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:31.200390 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:31.208580 (+ 8190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:31.208587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:31.208592 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:31.208633 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:31.212437 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.216581 (+ 4144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.216587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.216590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.216776 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:31.217262 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:31.217267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:31.217270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:31.217653 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:31.217895 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:31.218565 (+ 670us) server_negotiation.cc:299] Negotiation successful
0806 20:43:31.218825 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 20:43:31.240156 15407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7416b9eef51b4cca8b6097ae46eb5d3b: Generated new TSK 0
I20260806 20:43:31.242589 15407 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:31.270519 15435 catalog_manager.cc:1259] Loaded cluster ID: 6f4e35f97d8b4db8a23e22d602cfdbf2
I20260806 20:43:31.270730 15435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: loading cluster ID for follower catalog manager: success
I20260806 20:43:31.274719 15435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: acquiring CA information for follower catalog manager: success
W20260806 20:43:31.276844 15435 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260806 20:43:31.402638 15371 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.384629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51450 (local address 127.23.200.60:39059)
0806 20:43:31.387654 (+ 3025us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.387664 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.393142 (+ 5478us) server_negotiation.cc:407] Connection header received
0806 20:43:31.393341 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.393348 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.393435 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.393554 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.402375 (+ 8821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51450: BlockingRecv error: recv got EOF from 127.0.0.1:51450 (error 108)
Metrics: {"server-negotiator.queue_time_us":2839}
W20260806 20:43:31.405087 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.403138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.23.200.61:45597)
0806 20:43:31.403412 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.403455 (+ 43us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.403476 (+ 21us) server_negotiation.cc:407] Connection header received
0806 20:43:31.403632 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.403638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.403739 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.403840 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.404865 (+ 1025us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43756: BlockingRecv error: recv got EOF from 127.0.0.1:43756 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
W20260806 20:43:31.411962 15372 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.384145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60644 (local address 127.23.200.62:40651)
0806 20:43:31.384363 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.384370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.405688 (+ 21318us) server_negotiation.cc:407] Connection header received
0806 20:43:31.405795 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.405801 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.405879 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.405982 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.411711 (+ 5729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60644: BlockingRecv error: recv got EOF from 127.0.0.1:60644 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260806 20:43:31.468961 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.454763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43762 (local address 127.23.200.61:45597)
0806 20:43:31.454965 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.454972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.466910 (+ 11938us) server_negotiation.cc:407] Connection header received
0806 20:43:31.467203 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.467211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.467346 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.467440 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.468658 (+ 1218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43762: BlockingRecv error: recv got EOF from 127.0.0.1:43762 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 20:43:31.512564 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.503538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43766 (local address 127.23.200.61:45597)
0806 20:43:31.503730 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.503737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.510793 (+ 7056us) server_negotiation.cc:407] Connection header received
0806 20:43:31.511096 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.511102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.511204 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.511280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.512310 (+ 1030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43766: BlockingRecv error: recv got EOF from 127.0.0.1:43766 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260806 20:43:31.564723 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.552244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.23.200.61:45597)
0806 20:43:31.552457 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.552463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.562914 (+ 10451us) server_negotiation.cc:407] Connection header received
0806 20:43:31.563174 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.563180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.563305 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.563408 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.564435 (+ 1027us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43776: BlockingRecv error: recv got EOF from 127.0.0.1:43776 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260806 20:43:31.615648 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.605605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.23.200.61:45597)
0806 20:43:31.605861 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.605867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.613491 (+ 7624us) server_negotiation.cc:407] Connection header received
0806 20:43:31.614020 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.614030 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.614128 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.614207 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.615381 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43788: BlockingRecv error: recv got EOF from 127.0.0.1:43788 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
W20260806 20:43:31.666985 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.657855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.23.200.61:45597)
0806 20:43:31.658053 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.658059 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.664947 (+ 6888us) server_negotiation.cc:407] Connection header received
0806 20:43:31.665476 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.665483 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.665583 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.665668 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.666728 (+ 1060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43792: BlockingRecv error: recv got EOF from 127.0.0.1:43792 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 20:43:31.719318 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.716800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43806 (local address 127.23.200.61:45597)
0806 20:43:31.716992 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.717000 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.717619 (+ 619us) server_negotiation.cc:407] Connection header received
0806 20:43:31.717843 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.717849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.717947 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.718065 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.719039 (+ 974us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43806: BlockingRecv error: recv got EOF from 127.0.0.1:43806 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260806 20:43:31.759428 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.757290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43816 (local address 127.23.200.61:45597)
0806 20:43:31.757463 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.757468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.757871 (+ 403us) server_negotiation.cc:407] Connection header received
0806 20:43:31.758091 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.758095 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.758165 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.758259 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.759140 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43816: BlockingRecv error: recv got EOF from 127.0.0.1:43816 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260806 20:43:31.829749 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.827586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.23.200.61:45597)
0806 20:43:31.827875 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.827881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.828199 (+ 318us) server_negotiation.cc:407] Connection header received
0806 20:43:31.828456 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.828460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.828540 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.828626 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.829499 (+ 873us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43822: BlockingRecv error: recv got EOF from 127.0.0.1:43822 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260806 20:43:31.963146 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:31.961011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.23.200.61:45597)
0806 20:43:31.961273 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:31.961280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:31.961582 (+ 302us) server_negotiation.cc:407] Connection header received
0806 20:43:31.961820 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:31.961825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:31.961905 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:31.962018 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:31.962859 (+ 841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43832: BlockingRecv error: recv got EOF from 127.0.0.1:43832 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 20:43:31.998824 15386 catalog_manager.cc:1259] Loaded cluster ID: 6f4e35f97d8b4db8a23e22d602cfdbf2
I20260806 20:43:31.998952 15386 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: loading cluster ID for follower catalog manager: success
I20260806 20:43:32.001470 15386 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: acquiring CA information for follower catalog manager: success
I20260806 20:43:32.003366 15386 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 06598d7a2e6a415b9efaa6ad86a28492: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 20:43:32.224359 15296 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:32.222646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.23.200.61:45597)
0806 20:43:32.222835 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:32.222840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:32.223128 (+ 288us) server_negotiation.cc:407] Connection header received
0806 20:43:32.223301 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:32.223305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:32.223375 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:32.223464 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:32.224184 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43838: BlockingRecv error: recv got EOF from 127.0.0.1:43838 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:43:32.279726 15435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1011c68026224265af6886829080a18a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 20:43:32.747069 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:32.741085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.23.200.61:45597)
0806 20:43:32.741569 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:32.741573 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:32.741608 (+ 35us) server_negotiation.cc:407] Connection header received
0806 20:43:32.741820 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:32.741824 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:32.741991 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:32.742099 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:32.742564 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.743333 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.744775 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.745178 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.745270 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:32.745750 (+ 480us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:32.746405 (+ 655us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:32.746524 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:32.746650 (+ 126us) server_negotiation.cc:299] Negotiation successful
0806 20:43:32.746745 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":252,"threads_started":1}
I20260806 20:43:32.758793 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:32.753041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.23.200.60:39059)
0806 20:43:32.753513 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:32.753518 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:32.753552 (+ 34us) server_negotiation.cc:407] Connection header received
0806 20:43:32.753750 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:32.753753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:32.753900 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:32.754033 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:32.754408 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.755189 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.756535 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.756916 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.757008 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:32.757457 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:32.758110 (+ 653us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:32.758247 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:32.758381 (+ 134us) server_negotiation.cc:299] Negotiation successful
0806 20:43:32.758463 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260806 20:43:32.781857 15509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:32.769499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51474 (local address 127.23.200.60:39059)
0806 20:43:32.775773 (+ 6274us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:32.775781 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:32.776137 (+ 356us) server_negotiation.cc:407] Connection header received
0806 20:43:32.776519 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:32.776528 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:32.776655 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:32.776747 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:32.777446 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.778590 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.780211 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.780609 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.780738 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:32.781131 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:32.781222 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:32.781318 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:32.781492 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 20:43:32.781585 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6105}
I20260806 20:43:32.784181 15516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:32.769001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60658 (local address 127.23.200.62:40651)
0806 20:43:32.776542 (+ 7541us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:32.776550 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:32.776572 (+ 22us) server_negotiation.cc:407] Connection header received
0806 20:43:32.776694 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:32.776702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:32.776925 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:32.777144 (+ 219us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:32.778220 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.779124 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.781661 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.782258 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.782432 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:32.782927 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:32.783387 (+ 460us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:32.783498 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:32.783666 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 20:43:32.783748 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7387,"thread_start_us":7306,"threads_started":1}
I20260806 20:43:32.784336 15503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:32.769801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43848 (local address 127.23.200.61:45597)
0806 20:43:32.770059 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:32.770068 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:32.777046 (+ 6978us) server_negotiation.cc:407] Connection header received
0806 20:43:32.777274 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:32.777282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:32.777389 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:32.777457 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:32.778184 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.779172 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.782602 (+ 3430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:32.783090 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:32.783231 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:32.783624 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:32.783716 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:32.783811 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:32.783965 (+ 154us) server_negotiation.cc:299] Negotiation successful
0806 20:43:32.784044 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 20:43:32.790130 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15148
I20260806 20:43:32.791114 15211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:32.904435 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15148
I20260806 20:43:32.930151 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15223
I20260806 20:43:32.931308 15287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:33.038405 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15223
I20260806 20:43:33.059044 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15299
I20260806 20:43:33.060202 15362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:33.166277 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15299
2026-08-06T20:43:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5093 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 20:43:33 dist-test-slave-tm1g krb5kdc[15524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 20:43:33 dist-test-slave-tm1g krb5kdc[15524](info): set up 2 sockets
Aug 06 20:43:33 dist-test-slave-tm1g krb5kdc[15524](info): commencing operation
krb5kdc: starting...
W20260806 20:43:34.294250 24352 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/test-user.keytab.
Aug 06 20:43:34 dist-test-slave-tm1g krb5kdc[15524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T20:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T20:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.200.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
Entry for principal kudu/127.23.200.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab.
I20260806 20:43:34.561061 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:42409
--webserver_interface=127.23.200.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:42409,127.23.200.61:41595,127.23.200.60:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 20:43:34.888832 15537 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 20:43:34.889213 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:34.889339 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:34.889483 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 20:43:34.896757 15537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:34.896878 15537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:34.897011 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:34.897100 15537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:34.897212 15537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:34.906817 15537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:42409,127.23.200.61:41595,127.23.200.60:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.62.keytab
--principal=kudu/127.23.200.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.62:42409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.200.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:34.908198 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:34.909540 15537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:34.921802 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:34.921867 15544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:34.922396 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:34.922967 15537 server_base.cc:1034] running on GCE node
I20260806 20:43:34.923681 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:34.925021 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:34.926271 15537 hybrid_clock.cc:648] HybridClock initialized: now 1786049014926199 us; error 46 us; skew 500 ppm
I20260806 20:43:34.926620 15537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:34 dist-test-slave-tm1g krb5kdc[15524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:34.929821 15537 init.cc:377] Logged in from keytab as kudu/127.23.200.62@KRBTEST.COM (short username kudu)
I20260806 20:43:34.931208 15537 webserver.cc:460] Webserver started at http://127.23.200.62:34979/ using document root <none> and password file <none>
I20260806 20:43:34.931839 15537 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:34.931936 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:34.932193 15537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:34.935062 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data/instance:
uuid: "49124850c5d54e60a668e1826cfa623a"
format_stamp: "Formatted at 2026-08-06 20:43:34 on dist-test-slave-tm1g"
server_key: "8390b030589d734a52899caed5abbe1d"
server_key_iv: "43c4b4d08d78085f72a5b1c503f46604"
server_key_version: "encryptionkey@0"
I20260806 20:43:34.935778 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal/instance:
uuid: "49124850c5d54e60a668e1826cfa623a"
format_stamp: "Formatted at 2026-08-06 20:43:34 on dist-test-slave-tm1g"
server_key: "8390b030589d734a52899caed5abbe1d"
server_key_iv: "43c4b4d08d78085f72a5b1c503f46604"
server_key_version: "encryptionkey@0"
I20260806 20:43:34.940583 15537 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 20:43:34.943804 15553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:34.945048 15537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 20:43:34.945225 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal
uuid: "49124850c5d54e60a668e1826cfa623a"
format_stamp: "Formatted at 2026-08-06 20:43:34 on dist-test-slave-tm1g"
server_key: "8390b030589d734a52899caed5abbe1d"
server_key_iv: "43c4b4d08d78085f72a5b1c503f46604"
server_key_version: "encryptionkey@0"
I20260806 20:43:34.945441 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:34.975531 15537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:34.979298 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:34.979602 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:34.998041 15537 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.62:42409
I20260806 20:43:34.998054 15605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.62:42409 every 8 connection(s)
I20260806 20:43:34.999351 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/data/info.pb
I20260806 20:43:35.003034 15606 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:35.005352 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15537
I20260806 20:43:35.005553 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-0/wal/instance
I20260806 20:43:35.006137 24352 external_mini_cluster.cc:1428] Setting key a9ba9a1a72b7596078a3b684ff819437
I20260806 20:43:35.012236 15606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:35 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:35.037441 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.016039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42409 (local address 127.0.0.1:40342)
0806 20:43:35.016867 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.016916 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:35.019464 (+ 2548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:35.020378 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:35.020383 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:35.020913 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:35.021261 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.021278 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.023604 (+ 2326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.023608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.024654 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.024666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.025989 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.025994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.026110 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.026816 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:35.026837 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:35.029104 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:35.033560 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.033570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.033579 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.033964 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.034498 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.034502 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.034504 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.034654 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.035664 (+ 1010us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:35.035670 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:35.035844 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 20:43:35.036040 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 20:43:35.036434 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":349,"threads_started":1}
I20260806 20:43:35.037482 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.016039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40342 (local address 127.23.200.62:42409)
0806 20:43:35.017661 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:35.017668 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:35.017720 (+ 52us) server_negotiation.cc:407] Connection header received
0806 20:43:35.019824 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:35.019834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:35.020130 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:35.020323 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:35.021485 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.023455 (+ 1970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.024800 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.025766 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.025979 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.029345 (+ 3366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:35.029356 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:35.029372 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:35.029412 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:35.033435 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.034105 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.034117 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.034121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.034368 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.034768 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.034776 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.034782 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.035510 (+ 728us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:35.035697 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:35.036226 (+ 529us) server_negotiation.cc:299] Negotiation successful
0806 20:43:35.036599 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":1410,"spinlock_wait_cycles":16128,"thread_start_us":1184,"threads_started":1}
WARNING: no policy specified for kudu/127.23.200.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.61@KRBTEST.COM" created.
I20260806 20:43:35.040522 15606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:35.041509 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.041135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56426)
0806 20:43:35.041294 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.041401 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 20:43:35.041831 15555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
I20260806 20:43:35.043344 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.043037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56428)
0806 20:43:35.043165 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.043238 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 20:43:35.043635 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41595: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab.
Entry for principal kudu/127.23.200.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab.
I20260806 20:43:35.069206 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:41595
--webserver_interface=127.23.200.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab
--principal=kudu/127.23.200.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:42409,127.23.200.61:41595,127.23.200.60:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:43:35.092959 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } attempt: 1
I20260806 20:43:35.094192 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.093705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56438)
0806 20:43:35.093912 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.094038 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 20:43:35.095181 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.094841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56454)
0806 20:43:35.094991 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.095059 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 20:43:35.095465 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41595: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
I20260806 20:43:35.163722 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } attempt: 2
I20260806 20:43:35.165042 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.164558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56460)
0806 20:43:35.164802 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.164904 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 20:43:35.166100 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.165725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56466)
0806 20:43:35.165875 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.165964 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 20:43:35.166281 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41595: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
I20260806 20:43:35.257644 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } attempt: 3
I20260806 20:43:35.258875 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.258457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56472)
0806 20:43:35.258651 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.258743 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260806 20:43:35.259842 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.259500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56480)
0806 20:43:35.259649 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.259723 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 20:43:35.260023 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41595: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
W20260806 20:43:35.398020 15613 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 20:43:35.398404 15613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:35.398536 15613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:35.398675 15613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 20:43:35.403261 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } attempt: 4
I20260806 20:43:35.404577 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.403986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56494)
0806 20:43:35.404351 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.404440 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187}
I20260806 20:43:35.405575 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.405224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56502)
0806 20:43:35.405385 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.405457 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 20:43:35.405783 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.61:41595: Network error: Client connection negotiation failed: client connection to 127.23.200.61:41595: connect: Connection refused (error 111)
W20260806 20:43:35.406596 15613 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:35.406685 15613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:35.406790 15613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:35.406874 15613 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:35.406978 15613 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:35.416551 15613 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:42409,127.23.200.61:41595,127.23.200.60:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.61.keytab
--principal=kudu/127.23.200.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.61:41595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.200.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:35.417829 15613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:35.419239 15613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 20:43:35.431141 15618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:35.431183 15621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:35.431234 15619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:35.435756 15613 server_base.cc:1034] running on GCE node
I20260806 20:43:35.436532 15613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:35.437943 15613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:35.439196 15613 hybrid_clock.cc:648] HybridClock initialized: now 1786049015439132 us; error 55 us; skew 500 ppm
I20260806 20:43:35.439530 15613 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:35 dist-test-slave-tm1g krb5kdc[15524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:35.442711 15613 init.cc:377] Logged in from keytab as kudu/127.23.200.61@KRBTEST.COM (short username kudu)
I20260806 20:43:35.444278 15613 webserver.cc:460] Webserver started at http://127.23.200.61:44655/ using document root <none> and password file <none>
I20260806 20:43:35.445015 15613 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:35.445147 15613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:35.445489 15613 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:35.448529 15613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data/instance:
uuid: "9fe56a4cf603429680fe7ef874d8b8f9"
format_stamp: "Formatted at 2026-08-06 20:43:35 on dist-test-slave-tm1g"
server_key: "a4cad204ea45e4e962e74d0daa34c3af"
server_key_iv: "7cd89f00cf675863e35ff9b264a3ff4c"
server_key_version: "encryptionkey@0"
I20260806 20:43:35.449330 15613 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal/instance:
uuid: "9fe56a4cf603429680fe7ef874d8b8f9"
format_stamp: "Formatted at 2026-08-06 20:43:35 on dist-test-slave-tm1g"
server_key: "a4cad204ea45e4e962e74d0daa34c3af"
server_key_iv: "7cd89f00cf675863e35ff9b264a3ff4c"
server_key_version: "encryptionkey@0"
I20260806 20:43:35.454500 15613 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260806 20:43:35.457707 15628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:35.459045 15613 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 20:43:35.459177 15613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal
uuid: "9fe56a4cf603429680fe7ef874d8b8f9"
format_stamp: "Formatted at 2026-08-06 20:43:35 on dist-test-slave-tm1g"
server_key: "a4cad204ea45e4e962e74d0daa34c3af"
server_key_iv: "7cd89f00cf675863e35ff9b264a3ff4c"
server_key_version: "encryptionkey@0"
I20260806 20:43:35.459349 15613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:35.490180 15613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:35.494089 15613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:35.494329 15613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:35.512676 15613 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.61:41595
I20260806 20:43:35.512709 15679 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.61:41595 every 8 connection(s)
I20260806 20:43:35.513959 15613 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/data/info.pb
I20260806 20:43:35.517702 15680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:35.522500 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15613
I20260806 20:43:35.522689 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-1/wal/instance
I20260806 20:43:35.523115 24352 external_mini_cluster.cc:1428] Setting key 8ee0f82ec06fcec348cd6727801ee985
I20260806 20:43:35.524236 15680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:35 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:35.544953 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.526787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40348 (local address 127.23.200.62:42409)
0806 20:43:35.526975 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:35.526980 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:35.528177 (+ 1197us) server_negotiation.cc:407] Connection header received
0806 20:43:35.530048 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:35.530054 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:35.530148 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:35.530269 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:35.531717 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.532468 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.533966 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.534456 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.534591 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.537935 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:35.537945 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:35.537953 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:35.538002 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:35.541671 (+ 3669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.542439 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.542447 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.542453 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.542607 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.543008 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.543016 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.543022 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.543456 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:35.543614 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:35.544239 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 20:43:35.544506 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 20:43:35.545554 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.527303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42409 (local address 127.0.0.1:40348)
0806 20:43:35.528014 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.528057 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:35.529778 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:35.530524 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:35.530531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:35.531029 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:35.531539 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.531555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.532636 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.532640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.533805 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.533816 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.534685 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.534692 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.534924 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.535593 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:35.535614 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:35.537690 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:35.541819 (+ 4129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.541825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.541836 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.542264 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.542717 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.542720 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.542722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.542890 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.543577 (+ 687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:35.543582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:35.543864 (+ 282us) client_negotiation.cc:769] Sending connection context
0806 20:43:35.544096 (+ 232us) client_negotiation.cc:240] Negotiation successful
0806 20:43:35.544444 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":310,"threads_started":1}
I20260806 20:43:35.547335 15680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.200.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.200.60@KRBTEST.COM" created.
Aug 06 20:43:35 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:43:35.561555 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.547959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56518)
0806 20:43:35.548145 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.548162 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:35.548241 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:35.549443 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:35.549447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:35.549931 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:35.550061 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.550068 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.551525 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.551532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.552543 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.552554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.553310 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.553315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.553500 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.554038 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:35.554054 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:35.555979 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:35.559032 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.559039 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.559045 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.559384 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.559855 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.559860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.559863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.560022 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.560601 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:35.560608 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:35.560763 (+ 155us) client_negotiation.cc:769] Sending connection context
0806 20:43:35.560897 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 20:43:35.561178 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 20:43:35.561682 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.548346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56518 (local address 127.23.200.61:41595)
0806 20:43:35.548967 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:35.548973 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:35.548997 (+ 24us) server_negotiation.cc:407] Connection header received
0806 20:43:35.549085 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:35.549099 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:35.549303 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:35.549464 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:35.550238 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.551387 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.552670 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.553173 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.553305 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.556183 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:35.556193 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:35.556203 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:35.556229 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:35.558908 (+ 2679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.559512 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.559569 (+ 57us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.559572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.559707 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.560131 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.560138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.560141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.560471 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:35.560696 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:35.561046 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 20:43:35.561298 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":488,"thread_start_us":363,"threads_started":1}
I20260806 20:43:35.564056 15680 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:35.565227 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.564670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52072)
0806 20:43:35.564863 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.565018 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 20:43:35.565480 15630 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
I20260806 20:43:35.567080 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.566664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52082)
0806 20:43:35.566839 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.566940 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 20:43:35.567286 15680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.200.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab.
Entry for principal kudu/127.23.200.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab.
I20260806 20:43:35.583742 24352 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
/tmp/dist-test-taskG_AEDx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:41505
--webserver_interface=127.23.200.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab
--principal=kudu/127.23.200.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.200.20:46467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.200.62:42409,127.23.200.61:41595,127.23.200.60:41505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 20:43:35.616624 15680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 1
I20260806 20:43:35.617895 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.617362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52094)
0806 20:43:35.617580 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.617706 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260806 20:43:35.618963 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.618577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52098)
0806 20:43:35.618746 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.618837 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 20:43:35.619186 15680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
I20260806 20:43:35.687458 15680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 2
I20260806 20:43:35.688756 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.688210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52104)
0806 20:43:35.688450 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.688566 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 20:43:35.689888 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.689474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52120)
0806 20:43:35.689636 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.689720 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260806 20:43:35.690136 15680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
I20260806 20:43:35.705040 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } attempt: 5
Aug 06 20:43:35 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:43:35.719211 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.705770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56528)
0806 20:43:35.705953 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.705975 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:35.706088 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:35.706500 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:35.706504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:35.706866 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:35.707053 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.707060 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.708453 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.708457 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.709352 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:35.709372 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.710390 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.710397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:35.710578 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.711202 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:35.711218 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:35.713181 (+ 1963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:35.716991 (+ 3810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.716994 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.716997 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.717344 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.717731 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:35.717735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:35.717737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:35.717871 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:35.718512 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:35.718515 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:35.718656 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 20:43:35.718766 (+ 110us) client_negotiation.cc:240] Negotiation successful
0806 20:43:35.718969 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 20:43:35.719477 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.705865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.23.200.61:41595)
0806 20:43:35.706083 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:35.706090 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:35.706113 (+ 23us) server_negotiation.cc:407] Connection header received
0806 20:43:35.706204 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:35.706209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:35.706324 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:35.706459 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:35.707210 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.708310 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.709517 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:35.710176 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:35.710320 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:35.713386 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:35.713393 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:35.713397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:35.713436 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:35.716871 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.717468 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.717473 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.717477 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.717623 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:35.717976 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:35.717981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:35.717984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:35.718391 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:35.718532 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:35.718923 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 20:43:35.719168 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 20:43:35.720597 15606 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:35.721688 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.721268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52126)
0806 20:43:35.721468 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.721555 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 20:43:35.722693 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.722380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52142)
0806 20:43:35.722539 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.722602 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260806 20:43:35.722918 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
I20260806 20:43:35.774173 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 1
I20260806 20:43:35.775386 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.774913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52158)
0806 20:43:35.775119 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.775224 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260806 20:43:35.776412 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.776085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52170)
0806 20:43:35.776249 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.776313 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 20:43:35.776630 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
I20260806 20:43:35.781394 15680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 3
I20260806 20:43:35.782629 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.782012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52174)
0806 20:43:35.782255 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.782408 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 20:43:35.783830 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.783282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52180)
0806 20:43:35.783545 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.783656 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
W20260806 20:43:35.784117 15680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
I20260806 20:43:35.844854 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 2
I20260806 20:43:35.846002 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.845512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52182)
0806 20:43:35.845738 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.845845 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 20:43:35.847077 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.846752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52190)
0806 20:43:35.846905 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.846980 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 20:43:35.847295 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
W20260806 20:43:35.925590 15688 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 20:43:35.925976 15688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 20:43:35.926088 15688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 20:43:35.926215 15688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 20:43:35.927348 15680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 4
I20260806 20:43:35.928507 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.927964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52194)
0806 20:43:35.928208 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.928314 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 20:43:35.929522 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.929159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52210)
0806 20:43:35.929313 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.929397 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260806 20:43:35.929772 15680 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
W20260806 20:43:35.934172 15688 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 20:43:35.934249 15688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 20:43:35.934422 15688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 20:43:35.934509 15688 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 20:43:35.934600 15688 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 20:43:35.944350 15688 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.200.20:46467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.200.62:42409,127.23.200.61:41595,127.23.200.60:41505
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/krb5kdc/kudu_127.23.200.60.keytab
--principal=kudu/127.23.200.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.200.60:41505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.200.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 19:07:42 UTC on ccycloud.quasar-cuhmbo.root.comops.site
build id 4571034
ASAN enabled
I20260806 20:43:35.945562 15688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 20:43:35.947052 15688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 20:43:35.953509 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 3
I20260806 20:43:35.954588 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.954069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52226)
0806 20:43:35.954339 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.954432 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 20:43:35.955610 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:35.955271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52240)
0806 20:43:35.955453 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:35.955515 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 20:43:35.955852 15606 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.200.60:41505: Network error: Client connection negotiation failed: client connection to 127.23.200.60:41505: connect: Connection refused (error 111)
W20260806 20:43:35.958760 15694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:35.958829 15696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 20:43:35.958855 15693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 20:43:35.960844 15688 server_base.cc:1034] running on GCE node
I20260806 20:43:35.961613 15688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 20:43:35.963056 15688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 20:43:35.964282 15688 hybrid_clock.cc:648] HybridClock initialized: now 1786049015964219 us; error 30 us; skew 500 ppm
I20260806 20:43:35.964607 15688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 20:43:35 dist-test-slave-tm1g krb5kdc[15524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 20:43:35.967717 15688 init.cc:377] Logged in from keytab as kudu/127.23.200.60@KRBTEST.COM (short username kudu)
I20260806 20:43:35.969090 15688 webserver.cc:460] Webserver started at http://127.23.200.60:46123/ using document root <none> and password file <none>
I20260806 20:43:35.969794 15688 fs_manager.cc:359] Metadata directory not provided
I20260806 20:43:35.969930 15688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 20:43:35.970266 15688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 20:43:35.973176 15688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data/instance:
uuid: "6b72b4bf345c4cfa82905aa179fd28b5"
format_stamp: "Formatted at 2026-08-06 20:43:35 on dist-test-slave-tm1g"
server_key: "2cf928eead114212198729be6e67ef54"
server_key_iv: "409bf7538b5ebdd4e25fd613ee5a3e49"
server_key_version: "encryptionkey@0"
I20260806 20:43:35.973968 15688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal/instance:
uuid: "6b72b4bf345c4cfa82905aa179fd28b5"
format_stamp: "Formatted at 2026-08-06 20:43:35 on dist-test-slave-tm1g"
server_key: "2cf928eead114212198729be6e67ef54"
server_key_iv: "409bf7538b5ebdd4e25fd613ee5a3e49"
server_key_version: "encryptionkey@0"
I20260806 20:43:35.978947 15688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260806 20:43:35.982208 15703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 20:43:35.983501 15688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 20:43:35.983693 15688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data,/tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal
uuid: "6b72b4bf345c4cfa82905aa179fd28b5"
format_stamp: "Formatted at 2026-08-06 20:43:35 on dist-test-slave-tm1g"
server_key: "2cf928eead114212198729be6e67ef54"
server_key_iv: "409bf7538b5ebdd4e25fd613ee5a3e49"
server_key_version: "encryptionkey@0"
I20260806 20:43:35.983918 15688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 20:43:35.996049 15688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 20:43:36.000900 15688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 20:43:36.001148 15688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 20:43:36.021502 15688 rpc_server.cc:307] RPC server started. Bound to: 127.23.200.60:41505
I20260806 20:43:36.021528 15755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.200.60:41505 every 8 connection(s)
I20260806 20:43:36.022881 15688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/data/info.pb
I20260806 20:43:36.026609 15756 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 20:43:36.027547 24352 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu as pid 15688
I20260806 20:43:36.027733 24352 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskG_AEDx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786048651485794-24352-0/minicluster-data/master-2/wal/instance
I20260806 20:43:36.028304 24352 external_mini_cluster.cc:1428] Setting key 06d302c4873b683833ad0394444dc57e
I20260806 20:43:36.035862 15756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:36.052768 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.030263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52244 (local address 127.23.200.60:41505)
0806 20:43:36.031092 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.031103 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.031149 (+ 46us) server_negotiation.cc:407] Connection header received
0806 20:43:36.032793 (+ 1644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.032814 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.033185 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.033507 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.035290 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.036502 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.038260 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.039375 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.039540 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.044306 (+ 4766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.044317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.044329 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.044377 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.048236 (+ 3859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.049046 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.049082 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.049091 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.049281 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.049740 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.049745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.049749 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.050480 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.050790 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.051085 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.051508 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":591,"thread_start_us":417,"threads_started":1}
I20260806 20:43:36.054524 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.038425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40354 (local address 127.23.200.62:42409)
0806 20:43:36.038643 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.038652 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.039160 (+ 508us) server_negotiation.cc:407] Connection header received
0806 20:43:36.039358 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.039364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.039462 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.039566 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.040612 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.041777 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.043137 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.043850 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.043998 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.048268 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.048277 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.048285 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.048331 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.051706 (+ 3375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.052397 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.052403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.052408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.052574 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.052981 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.052987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.052990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.053251 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.053360 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.053861 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.054112 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260806 20:43:36.054564 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.038358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.62:42409 (local address 127.0.0.1:40354)
0806 20:43:36.039027 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:36.039064 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:36.039226 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:36.039636 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:36.039643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:36.040228 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:36.040457 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.040465 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.041930 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.041933 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.042979 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.042991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.044064 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.044069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.044256 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.044959 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:36.044990 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:36.048069 (+ 3079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:36.051834 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.051843 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.051847 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.052261 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.052689 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.052692 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.052695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.052833 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.053435 (+ 602us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:36.053440 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:36.053580 (+ 140us) client_negotiation.cc:769] Sending connection context
0806 20:43:36.053749 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 20:43:36.054080 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":341,"threads_started":1}
I20260806 20:43:36.056483 15756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.62@KRBTEST.COM
I20260806 20:43:36.069885 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.057004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.61:41595 (local address 127.0.0.1:56534)
0806 20:43:36.057147 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:36.057163 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:36.057265 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:36.057769 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:36.057776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:36.058206 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:36.058419 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.058430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.059396 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.059404 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.060338 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.060368 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.061094 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.061100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.061277 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.061755 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:36.061770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:36.063961 (+ 2191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:36.067243 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.067248 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.067251 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.067743 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.068095 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.068100 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.068104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.068290 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.068971 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:36.068975 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:36.069135 (+ 160us) client_negotiation.cc:769] Sending connection context
0806 20:43:36.069313 (+ 178us) client_negotiation.cc:240] Negotiation successful
0806 20:43:36.069535 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260806 20:43:36.069960 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.057350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56534 (local address 127.23.200.61:41595)
0806 20:43:36.057531 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.057536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.057550 (+ 14us) server_negotiation.cc:407] Connection header received
0806 20:43:36.057596 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.057600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.057654 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.057726 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.058579 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.059262 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.060518 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.060945 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.061074 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.064182 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.064188 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.064194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.064235 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.067117 (+ 2882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.067853 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.067856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.067858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.067985 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.068404 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.068408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.068411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.068846 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.069003 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.069449 (+ 446us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.069679 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 20:43:36.070938 15756 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } has no permanent_uuid. Determining permanent_uuid...
I20260806 20:43:36.071465 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.055870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40366 (local address 127.23.200.62:42409)
0806 20:43:36.056034 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.056041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.056058 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:43:36.056226 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.056231 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.056294 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.056391 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.057690 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.058739 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.060016 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.060476 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.060626 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.065189 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.065194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.065197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.065225 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.068278 (+ 3053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.069081 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.069086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.069090 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.069232 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.069645 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.069650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.069654 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.069996 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.070253 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.070938 (+ 685us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.071151 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.60@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.200.61@KRBTEST.COM
I20260806 20:43:36.084215 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.071789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52256 (local address 127.23.200.60:41505)
0806 20:43:36.071942 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.071947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.071963 (+ 16us) server_negotiation.cc:407] Connection header received
0806 20:43:36.072057 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.072062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.072125 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.072216 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.073007 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.073903 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.075328 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.075812 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.075939 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.079039 (+ 3100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.079044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.079047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.079072 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.081574 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.082146 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.082153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.082160 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.082331 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.082795 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.082798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.082800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.083058 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.083185 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.083628 (+ 443us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.083981 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":59}
I20260806 20:43:36.084295 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.071616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52256)
0806 20:43:36.071846 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:36.071866 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:36.071967 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:36.072233 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:36.072237 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:36.072697 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:36.072882 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.072894 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.074021 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.074028 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.075190 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.075204 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.075957 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.075961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.076108 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.076612 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:36.076627 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:36.078851 (+ 2224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:36.081693 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.081700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.081704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.082031 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.082479 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.082483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.082502 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.082677 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.083181 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:36.083187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:36.083346 (+ 159us) client_negotiation.cc:769] Sending connection context
0806 20:43:36.083540 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 20:43:36.083844 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 20:43:36.087553 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.073117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56548 (local address 127.23.200.61:41595)
0806 20:43:36.073289 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.073295 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.073312 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:43:36.073364 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.073369 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.073430 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.073527 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.074648 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.075594 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.077021 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.077731 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.077853 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.081772 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.081778 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.081782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.081820 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.084911 (+ 3091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.085603 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.085610 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.085613 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.085745 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.086122 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.086127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.086130 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.086472 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.086751 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.087000 (+ 249us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.087212 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 20:43:36.089709 15756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: Bootstrap starting.
I20260806 20:43:36.092063 24352 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 20:43:36.093161 15756 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:36.094218 15756 log.cc:824] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:36.097404 15756 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: No bootstrap required, opened a new log
I20260806 20:43:36.094435 15580 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "49124850c5d54e60a668e1826cfa623a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40366
I20260806 20:43:36.108080 15756 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.108457 15756 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b72b4bf345c4cfa82905aa179fd28b5, State: Initialized, Role: FOLLOWER
I20260806 20:43:36.108943 15756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.109798 15762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.110132 15762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.111022 15756 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: configured and running, proceeding with master startup.
W20260806 20:43:36.120085 15773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:43:36.120239 15773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:43:36.118048 15688 master_runner.cc:186] Error getting master registration for 127.23.200.62:42409: OK, instance_id { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" instance_seqno: 1786049014992770 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 20:43:36.125862 15688 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 20:43:36.133064 15606 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 4
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.62@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
I20260806 20:43:36.147555 15608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.133659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52264)
0806 20:43:36.133857 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:36.133878 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:36.134007 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:36.134405 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:36.134410 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:36.134820 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:36.134986 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.134994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.136030 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.136036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.136992 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.137003 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.137773 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.137780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.138028 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.138649 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:36.138665 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:36.140697 (+ 2032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:36.144652 (+ 3955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.144657 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.144661 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.145173 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.145577 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.145582 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.145586 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.145762 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.146515 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:36.146521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:36.146703 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 20:43:36.146884 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 20:43:36.147144 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260806 20:43:36.147619 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.133713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52264 (local address 127.23.200.60:41505)
0806 20:43:36.133891 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.133899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.133924 (+ 25us) server_negotiation.cc:407] Connection header received
0806 20:43:36.134139 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.134144 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.134219 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.134350 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.135124 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.135881 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.137140 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.137631 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.137763 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.140896 (+ 3133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.140903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.140908 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.140954 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.144475 (+ 3521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.145314 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.145319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.145323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.145461 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.145882 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.145888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.145891 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.146369 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.146547 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.147008 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.147232 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260806 20:43:36.153425 15606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: Bootstrap starting.
I20260806 20:43:36.157258 15606 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:36.158550 15606 log.cc:824] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:36.161357 15606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: No bootstrap required, opened a new log
I20260806 20:43:36.168678 15606 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.169063 15606 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49124850c5d54e60a668e1826cfa623a, State: Initialized, Role: FOLLOWER
I20260806 20:43:36.169543 15606 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.170521 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.170928 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.171682 15606 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:36.171799 15580 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 20:43:36.171986 15580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:36.175951 15580 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.179720 15730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49124850c5d54e60a668e1826cfa623a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6b72b4bf345c4cfa82905aa179fd28b5"
I20260806 20:43:36.180222 15730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:36.183753 15730 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49124850c5d54e60a668e1826cfa623a in term 1.
I20260806 20:43:36.184370 15580 leader_election.cc:288] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [CANDIDATE]: Term 1 election: Requested vote from peers 9fe56a4cf603429680fe7ef874d8b8f9 (127.23.200.61:41595), 6b72b4bf345c4cfa82905aa179fd28b5 (127.23.200.60:41505)
I20260806 20:43:36.184383 15555 leader_election.cc:302] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49124850c5d54e60a668e1826cfa623a, 6b72b4bf345c4cfa82905aa179fd28b5; no voters:
I20260806 20:43:36.185051 15776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 1 FOLLOWER]: Leader election won for term 1
W20260806 20:43:36.188129 15787 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 20:43:36.194419 15787 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 20:43:36.194416 15776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [term 1 LEADER]: Becoming Leader. State: Replica: 49124850c5d54e60a668e1826cfa623a, State: Running, Role: LEADER
I20260806 20:43:36.195065 15776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.186906 15537 master_runner.cc:186] Error getting master registration for 127.23.200.61:41595: OK, instance_id { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" instance_seqno: 1786049015507491 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 20:43:36.199589 15537 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 20:43:36.203076 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49124850c5d54e60a668e1826cfa623a. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.203584 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:36.205940 15797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 20:43:36.211344 15797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 20:43:36.220839 15609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.201327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40374 (local address 127.23.200.62:42409)
0806 20:43:36.201529 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.201537 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.202659 (+ 1122us) server_negotiation.cc:407] Connection header received
0806 20:43:36.202943 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.202948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.203029 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.203105 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.205573 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.206845 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.208830 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.209555 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.209658 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.213269 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.213276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.213281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.213319 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.217105 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.217862 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.217868 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.217872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.218062 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.218552 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.218558 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.218561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.218948 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.219195 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.220226 (+ 1031us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.220467 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 20:43:36.228998 15680 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } attempt: 5
W20260806 20:43:36.232434 15559 tablet.cc:2369] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:43:36.235582 15730 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [term 1 FOLLOWER]: Refusing update from remote peer 49124850c5d54e60a668e1826cfa623a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:43:36.236696 15776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:43:36.238406 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.214605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52278 (local address 127.23.200.60:41505)
0806 20:43:36.214835 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.214842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.215690 (+ 848us) server_negotiation.cc:407] Connection header received
0806 20:43:36.215949 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.215954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.216041 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.216111 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.222170 (+ 6059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.223331 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.225365 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.226070 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.226180 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.229475 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.229481 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.229486 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.229526 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.233361 (+ 3835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.235843 (+ 2482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.235849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.235852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.236024 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.236468 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.236475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.236480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.236830 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.237207 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.237757 (+ 550us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.238010 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 20:43:36.241756 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.202804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56550 (local address 127.23.200.61:41595)
0806 20:43:36.203010 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.203018 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.214109 (+ 11091us) server_negotiation.cc:407] Connection header received
0806 20:43:36.218413 (+ 4304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.218419 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.218506 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.218583 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.220685 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.221801 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.228541 (+ 6740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.229202 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.229301 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.234066 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.234072 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.234076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.234115 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.237723 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.238539 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.238544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.238547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.238724 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.239774 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.239781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.239786 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.240300 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.240521 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.241147 (+ 626us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.241376 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Aug 06 20:43:36 dist-test-slave-tm1g krb5kdc[15524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786049015, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.200.61@KRBTEST.COM for kudu/127.23.200.60@KRBTEST.COM
W20260806 20:43:36.253302 15709 tablet.cc:2369] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 20:43:36.266211 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.229532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.200.60:41505 (local address 127.0.0.1:52288)
0806 20:43:36.230486 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0806 20:43:36.230508 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 20:43:36.230609 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 20:43:36.241706 (+ 11097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 20:43:36.241712 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 20:43:36.242181 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 20:43:36.242401 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.242412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.243930 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.243935 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.245336 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 20:43:36.245349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.246549 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.246554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 20:43:36.246792 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.247376 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 20:43:36.247400 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 20:43:36.250092 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 20:43:36.260190 (+ 10098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.260230 (+ 40us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.260249 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.261409 (+ 1160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.262605 (+ 1196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 20:43:36.262612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 20:43:36.262618 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 20:43:36.263023 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 20:43:36.264554 (+ 1531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 20:43:36.264559 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 20:43:36.264784 (+ 225us) client_negotiation.cc:769] Sending connection context
0806 20:43:36.265054 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 20:43:36.265575 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277}
I20260806 20:43:36.268916 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:36.230094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52288 (local address 127.23.200.60:41505)
0806 20:43:36.238850 (+ 8756us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.238856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.238873 (+ 17us) server_negotiation.cc:407] Connection header received
0806 20:43:36.238929 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.238935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.239007 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.239105 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.242659 (+ 3554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.243673 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.245478 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:36.246406 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:36.246528 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:36.250418 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 20:43:36.250429 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 20:43:36.250437 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 20:43:36.250551 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0806 20:43:36.259920 (+ 9369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.261771 (+ 1851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.261779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.261785 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.262209 (+ 424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 20:43:36.263317 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 20:43:36.263396 (+ 79us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 20:43:36.263438 (+ 42us) server_negotiation.cc:967] calling sasl_server_step()
0806 20:43:36.264299 (+ 861us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 20:43:36.266926 (+ 2627us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:36.267276 (+ 350us) server_negotiation.cc:299] Negotiation successful
0806 20:43:36.267715 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8650,"threads_started":1}
I20260806 20:43:36.273563 15762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49124850c5d54e60a668e1826cfa623a. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.273921 15762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.275117 15799 mvcc.cc:205] Tried to move back new op lower bound from 7315656770467078144 to 7315656770340741120. Current Snapshot: MvccSnapshot[applied={T|T < 7315656770467078144}]
I20260806 20:43:36.275527 15776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.275820 15776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:36.276952 15762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.277351 15762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.277554 15801 mvcc.cc:205] Tried to move back new op lower bound from 7315656770467078144 to 7315656770340741120. Current Snapshot: MvccSnapshot[applied={T|T < 7315656770467078144}]
I20260806 20:43:36.282559 15797 catalog_manager.cc:1347] Generated new cluster ID: 4434f2cb3c0043709045bbf64acad86f
I20260806 20:43:36.282655 15797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 20:43:36.284375 15680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: Bootstrap starting.
I20260806 20:43:36.288297 15680 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: Neither blocks nor log segments found. Creating new log.
I20260806 20:43:36.289505 15680 log.cc:824] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: Log is configured to *not* fsync() on all Append() calls
I20260806 20:43:36.293484 15680 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: No bootstrap required, opened a new log
I20260806 20:43:36.305485 15680 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.305876 15680 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fe56a4cf603429680fe7ef874d8b8f9, State: Initialized, Role: FOLLOWER
I20260806 20:43:36.306399 15680 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } }
I20260806 20:43:36.314888 15680 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 20:43:36.314958 15655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "49124850c5d54e60a668e1826cfa623a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9fe56a4cf603429680fe7ef874d8b8f9"
I20260806 20:43:36.315408 15655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 20:43:36.319478 15655 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 49124850c5d54e60a668e1826cfa623a in term 1.
I20260806 20:43:36.320402 15654 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [term 1 FOLLOWER]: Refusing update from remote peer 49124850c5d54e60a668e1826cfa623a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 20:43:36.321249 15789 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 20:43:36.325886 15806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.326231 15806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.363655 15806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49124850c5d54e60a668e1826cfa623a. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.363943 15806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.364270 15808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.364575 15808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 20:43:36.397161 15789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "49124850c5d54e60a668e1826cfa623a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49124850c5d54e60a668e1826cfa623a" member_type: VOTER last_known_addr { host: "127.23.200.62" port: 42409 } } peers { permanent_uuid: "9fe56a4cf603429680fe7ef874d8b8f9" member_type: VOTER last_known_addr { host: "127.23.200.61" port: 41595 } } peers { permanent_uuid: "6b72b4bf345c4cfa82905aa179fd28b5" member_type: VOTER last_known_addr { host: "127.23.200.60" port: 41505 } } }
I20260806 20:43:36.397433 15789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a [sys.catalog]: This master's current role is: LEADER
I20260806 20:43:36.407809 15797 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 20:43:36.410226 15797 catalog_manager.cc:1504] Loading token signing keys...
I20260806 20:43:36.433259 15797 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49124850c5d54e60a668e1826cfa623a: Generated new TSK 0
I20260806 20:43:36.434326 15797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 20:43:36.437041 15821 catalog_manager.cc:1259] Loaded cluster ID: 4434f2cb3c0043709045bbf64acad86f
I20260806 20:43:36.437135 15821 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: loading cluster ID for follower catalog manager: success
I20260806 20:43:36.440685 15821 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: acquiring CA information for follower catalog manager: success
W20260806 20:43:36.442026 15821 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
W20260806 20:43:36.601205 15609 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.588741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40390 (local address 127.23.200.62:42409)
0806 20:43:36.588975 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.588982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.597983 (+ 9001us) server_negotiation.cc:407] Connection header received
0806 20:43:36.599485 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.599496 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.599598 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.599692 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.600928 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40390: BlockingRecv error: recv got EOF from 127.0.0.1:40390 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 20:43:36.604837 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.598612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.23.200.61:41595)
0806 20:43:36.598841 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.598847 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.602651 (+ 3804us) server_negotiation.cc:407] Connection header received
0806 20:43:36.603420 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.603427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.603517 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.603593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.604583 (+ 990us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56558: BlockingRecv error: recv got EOF from 127.0.0.1:56558 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260806 20:43:36.610538 15800 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.607651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52298 (local address 127.23.200.60:41505)
0806 20:43:36.607818 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.607823 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.608393 (+ 570us) server_negotiation.cc:407] Connection header received
0806 20:43:36.608652 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.608663 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.608745 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.608828 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.610120 (+ 1292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52298: BlockingRecv error: recv got EOF from 127.0.0.1:52298 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260806 20:43:36.656008 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.641288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56572 (local address 127.23.200.61:41595)
0806 20:43:36.641487 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.641496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.653727 (+ 12231us) server_negotiation.cc:407] Connection header received
0806 20:43:36.654279 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.654286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.654449 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.654534 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.655677 (+ 1143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56572: BlockingRecv error: recv got EOF from 127.0.0.1:56572 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260806 20:43:36.699659 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.687514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.23.200.61:41595)
0806 20:43:36.687730 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.687739 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.697539 (+ 9800us) server_negotiation.cc:407] Connection header received
0806 20:43:36.698092 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.698099 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.698194 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.698282 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.699411 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56584: BlockingRecv error: recv got EOF from 127.0.0.1:56584 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 20:43:36.747700 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.735100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56588 (local address 127.23.200.61:41595)
0806 20:43:36.735278 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.735284 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.745908 (+ 10624us) server_negotiation.cc:407] Connection header received
0806 20:43:36.746418 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.746422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.746518 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.746649 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.747435 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56588: BlockingRecv error: recv got EOF from 127.0.0.1:56588 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260806 20:43:36.757763 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.755557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.23.200.61:41595)
0806 20:43:36.755829 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.755837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.756377 (+ 540us) server_negotiation.cc:407] Connection header received
0806 20:43:36.756590 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.756600 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.756667 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.756782 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.757526 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56604: BlockingRecv error: recv got EOF from 127.0.0.1:56604 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
W20260806 20:43:36.771605 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.769151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56620 (local address 127.23.200.61:41595)
0806 20:43:36.769333 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.769342 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.770208 (+ 866us) server_negotiation.cc:407] Connection header received
0806 20:43:36.770474 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.770483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.770571 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.770667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.771357 (+ 690us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56620: BlockingRecv error: recv got EOF from 127.0.0.1:56620 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260806 20:43:36.792863 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.790875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.23.200.61:41595)
0806 20:43:36.791088 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.791096 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.791455 (+ 359us) server_negotiation.cc:407] Connection header received
0806 20:43:36.791683 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.791687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.791755 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.791883 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.792609 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56622: BlockingRecv error: recv got EOF from 127.0.0.1:56622 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260806 20:43:36.830274 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.828204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56638 (local address 127.23.200.61:41595)
0806 20:43:36.828431 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.828437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.828823 (+ 386us) server_negotiation.cc:407] Connection header received
0806 20:43:36.829001 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.829007 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.829115 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.829238 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.829971 (+ 733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56638: BlockingRecv error: recv got EOF from 127.0.0.1:56638 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 20:43:36.899674 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:36.897713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.23.200.61:41595)
0806 20:43:36.897912 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:36.897926 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:36.898248 (+ 322us) server_negotiation.cc:407] Connection header received
0806 20:43:36.898497 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:36.898503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:36.898592 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:36.898705 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:36.899455 (+ 750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56640: BlockingRecv error: recv got EOF from 127.0.0.1:56640 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260806 20:43:37.033923 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:37.031807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.23.200.61:41595)
0806 20:43:37.032023 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.032029 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.032434 (+ 405us) server_negotiation.cc:407] Connection header received
0806 20:43:37.032690 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.032695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.032783 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.032896 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:37.033662 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56654: BlockingRecv error: recv got EOF from 127.0.0.1:56654 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 20:43:37.122152 15773 catalog_manager.cc:1259] Loaded cluster ID: 4434f2cb3c0043709045bbf64acad86f
I20260806 20:43:37.122248 15773 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: loading cluster ID for follower catalog manager: success
I20260806 20:43:37.124745 15773 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: acquiring CA information for follower catalog manager: success
I20260806 20:43:37.126552 15773 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b72b4bf345c4cfa82905aa179fd28b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 20:43:37.297951 15685 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 20:43:37.296134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.23.200.61:41595)
0806 20:43:37.296389 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.296396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.296646 (+ 250us) server_negotiation.cc:407] Connection header received
0806 20:43:37.296819 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.296823 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.296898 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.296988 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 20:43:37.297734 (+ 746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56664: BlockingRecv error: recv got EOF from 127.0.0.1:56664 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 20:43:37.445097 15821 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fe56a4cf603429680fe7ef874d8b8f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 20:43:37.819272 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:37.813579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.23.200.61:41595)
0806 20:43:37.814187 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.814195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.814230 (+ 35us) server_negotiation.cc:407] Connection header received
0806 20:43:37.814349 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.814372 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.814565 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.814674 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:37.815153 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.815871 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.817166 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.817573 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.817666 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:37.818117 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:37.818648 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:37.818758 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:37.818845 (+ 87us) server_negotiation.cc:299] Negotiation successful
0806 20:43:37.818910 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":255,"threads_started":1}
I20260806 20:43:37.830021 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:37.824455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.23.200.60:41505)
0806 20:43:37.824883 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.824887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.825168 (+ 281us) server_negotiation.cc:407] Connection header received
0806 20:43:37.825362 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.825366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.825520 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.825658 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:37.826022 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.826793 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.828090 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.828495 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.828605 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:37.829022 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:37.829486 (+ 464us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:37.829592 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:37.829685 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 20:43:37.829746 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":237,"threads_started":1}
I20260806 20:43:37.843073 15889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:37.836140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56672 (local address 127.23.200.61:41595)
0806 20:43:37.836313 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.836318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.836860 (+ 542us) server_negotiation.cc:407] Connection header received
0806 20:43:37.837136 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.837142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.837230 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.837294 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:37.838108 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.839003 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.841055 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.841530 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.841637 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:37.842371 (+ 734us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:37.842475 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:37.842578 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:37.842716 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 20:43:37.842801 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 20:43:37.843312 15901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:37.835474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40394 (local address 127.23.200.62:42409)
0806 20:43:37.835927 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.835935 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.836082 (+ 147us) server_negotiation.cc:407] Connection header received
0806 20:43:37.836400 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.836410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.836659 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.836833 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:37.837538 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.838722 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.840789 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.841449 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.841628 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:37.842169 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:37.842610 (+ 441us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:37.842710 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:37.842929 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 20:43:37.843011 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":243,"threads_started":1}
I20260806 20:43:37.843461 15895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 20:43:37.836932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52318 (local address 127.23.200.60:41505)
0806 20:43:37.837135 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0806 20:43:37.837143 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 20:43:37.837738 (+ 595us) server_negotiation.cc:407] Connection header received
0806 20:43:37.837999 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 20:43:37.838002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 20:43:37.838108 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 20:43:37.838203 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 20:43:37.838792 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.839577 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.841398 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 20:43:37.842030 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 20:43:37.842157 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 20:43:37.842698 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 20:43:37.842808 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 20:43:37.842895 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 20:43:37.843105 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 20:43:37.843180 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 20:43:37.848006 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15537
I20260806 20:43:37.849120 15601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:37.958793 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15537
W20260806 20:43:37.980284 15705 connection.cc:535] server connection from 127.0.0.1:52264 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 20:43:37.980691 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15613
I20260806 20:43:37.981726 15675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:38.089182 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15613
I20260806 20:43:38.110582 24352 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu pid 15688
I20260806 20:43:38.111842 15751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 20:43:38.218720 24352 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskG_AEDx/build/asan/bin/kudu with pid 15688
2026-08-06T20:43:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5052 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10146 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366700 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST